Memcached单键超1M数据量的拆分设计及测试

最新推荐文章于 2020-08-31 16:28:22 发布

原创最新推荐文章于 2020-08-31 16:28:22 发布 · 957 阅读

0 ·

本内容遵循CC 4.0 BY-SA版权协议

C# 专栏收录该内容

270 篇文章

订阅专栏

本文介绍了一种针对Memcached单个键值数据量超出1MB限制时的优化方案，通过将大型对象拆分为多个键值对来解决，并提供了一个具体的示例数据结构及其序列化压缩后的大小评估。

Memcached单个键值数据量不能超出默认的最大1M限制，这个时候需要进行拆分，成多个键。评估的时候需要实现模拟目标对象的数据量。比如我的数据结构如下：

        private static DTO.OnlineUser Makeuser(string key)
        {
            DTO.OnlineUser user = new DTO.OnlineUser();
            user.cachetag = key;
            user.userid = 123456;
            user.username = "jon";
            user.nickname = "土豆哥哥";
            user.gender = 0;
            user.qq = "*********";
            user.mobile = "1******6***";
            user.avatar = "/upload/user/img/20170424205344.jpg";
            user.createtime = DateTime.Now;
            user.money = 225.33m;
            user.videotime = 10800;
            user.recommendurl = "http://xxxxxx.xxxxxxxx.com/321453";
            user.issleep = false;
            user.isgag = false;
            user.attributor = "xxx-客服经理";
            user.allocationtime = DateTime.Now;
            user.logininfo = new DTO.logininfo
            {
                guid = Guid.NewGuid().ToString(),
                beattime = DateTime.Now,
                loginip = "192.168.20.45",
                loginaddress = "浙江省杭州市",
                logintime = DateTime.Now,
                lastloginip = "192.168.20.77",
                lastloginaddress = "安徽省合肥市",
                lastlogintime = DateTime.Now.AddDays(-7),
                logincount = 14
            };
            user.roleinfo = new DTO.roleinfo
            {
                id = 1,
                name = "金牌会员",
                type = 1,
                category = (int)DTO.category.客服经理,
                icon = "/upload/user/img/201704242103445.jpg",
                order = 2,
                goldadd = 20
            };
            return user;
        }

序列化压缩存储的时候，100个对象是83.5kb，那么可以预估，最多不能超出1200个，83.5*12=1002kb<1024kb；

同时考虑随机分配，随机算法取余（divisor = 16）如下：

            byte[] buffer = Guid.NewGuid().ToByteArray();
            var gid = BitConverter.ToInt64(buffer, 0);
            return gid % divisor;

分配的大致demo测试：

        private static void simulation()
        {            
            var temp = new List<Tuple<long, int>>();
            for (int i = 0; i < 16000; i++)
            {
                long key = GuidToLongID();//0~15
                temp.Add(new Tuple<long, int>(key, 1));
            }


            var t = temp.GroupBy(x => x.Item1).Select(x => new
            {
                k = x.FirstOrDefault().Item1,
                c = x.Sum(i => i.Item2)
            }).OrderBy(x => x.k);

            //Console.ForegroundColor = ConsoleColor.Green;
            //foreach (var i in t)
            //{
            //    Console.WriteLine("键：{0}，值：{1}", i.k, i.c);
            //}

            Console.ForegroundColor = ConsoleColor.White;
            var count = t.Max(x => x.c);
            var pp = t.Where(x => x.c == count).FirstOrDefault().k;
            Console.WriteLine("最大键：{0}，最大值：{1}", pp, count);

            Console.WriteLine("--------------------------------");
        }