不懂程序猿的世界 尤其是支付宝技术团队的程序猿
本以为程序猿已经是世界上最奇怪的工种,格子衬衫、随身带键盘、背双肩包、还秃顶,没想到,还有一种有“自虐”倾向的程序猿物种——支付宝技术团队。
对于他们而言,分成红蓝两队阵营,通过仿真环境模拟天灾人祸,上演“矛盾”之战,比网吧团战开黑来的更刺激。
从2016年开始,支付宝内部建立“红蓝对抗”机制,由蓝军寻找系统应用及架构上的脆弱点,牵引红军升级防控能力,每周“突袭”外加一年两大考,要求技术小哥哥们必须时刻绷紧神经。通过这种全栈级别的大型技术攻防演练,来检验并提升技术团队应急处理能力和风险意识,以进一步把控技术风险。
在这场技术攻防演练中,支付宝技术团队的程序猿们“玩”得不亦乐乎。蓝军成立两个月后,便推出了字节码级别的故障注入系统Awatch,实时侵入运行中的业务系统。同时,红军也积极建设防控体系,针对性地推出实时核对平台,做到稳定的分钟级核对异常发现能力。另外,各个业务域也研发了相应的核对系统,共同建立起稳固防线。
但由于大部分的故障场景主要是各个业务方提供的,更深层的隐患仍未被发现,蓝军意识到必须得自己主动去“找事儿”,于是在今年3月推出故障场景挖掘平台,最高能够在5分钟内产生500+的故障场景。
持续应急压力下,红军的程序猿们开展“故障自愈”架构体系升级及能力建设,配合蓝军研发了“无损”攻防体系以及相匹配的度量平台,让这场“攻防游戏”的结果数据可视化。
目前,对抗赛已经成为常态,保持每周200+故障场景的节奏运作。蓝军实时、随地、无差别地攻击,确实让红军的程序猿们费了不少心思。有趣的是,红军会在演练前举办拜关公的仪式,以求关公庇佑,还曾在一次周末突袭中感概“这次预期外的故障攻击是最真实的应急锻炼!”
从最开始简单设立技术质量团队和运维团队的青铜时代,到组建SRE团队(Site Reliability Engineer)的钻石时代,支付宝系统的稳定性逐步进阶,蚂蚁金服的技术风险也迅速发展,红蓝军对抗赛起到了关键性作用。 未来,蚂蚁金服SRE团队希望在技术风险防控方面做到5个9的能力,即99.999%,且任意故障在5分钟之内解决,并让技术风险防控具备智能特性,减少人工干预。
经过三年实践,全栈级技术攻防演练已经沉淀出一整套成熟的风险防控体系。现在正式对外开放,包括容灾应急平台、全链路压测、资金安全监控、变更管控、巡检平台以及黑屏运维管控等产品,已在蚂蚁金融科技官网(tech.antfin.com)逐步上线。
{"weixin":{"label":"微信","name":"weixin","selected":true,"value":true,"sortid":"1","shareid":"weixin","sharetitle":"分享到微信","event":"shareToWeiXin","lang":"shareWeb_WeiXin"},"copy":{"label":"复制网址","name":"copy","selected":true,"value":true,"sortid":"2","shareid":"copy","sharetitle":"复制网址","event":"copy_url","lang":"shareWeb_Copy"},"qq":{"label":"QQ好友","name":"qq","selected":true,"value":false,"sortid":"1","shareid":"qq","sharetitle":"分享到QQ","event":"shareToQQ","lang":"shareWeb_QQ"},"sina_weibo":{"label":"新浪微博","name":"sina_weibo","selected":true,"value":true,"sortid":"4","shareid":"sina_weibo","sharetitle":"分享到新浪微博","event":"shareToSinaWB","lang":"shareWeb_SinaWeiBo"},"qq_zone":{"label":"QQ空间","name":"qq_zone","selected":true,"value":true,"sortid":"5","shareid":"qq_zone","sharetitle":"分享到QQ空间","event":"shareToQzone","lang":"shareWeb_QQZone"},"renren":{"label":"人人网","name":"renren","selected":true,"value":true,"sortid":"7","shareid":"renren","sharetitle":"分享到人人网","event":"shareToRenren","lang":"shareWeb_RenRen"},"douban":{"label":"豆瓣网","name":"douban","selected":true,"value":true,"sortid":"8","shareid":"douban","sharetitle":"分享到豆瓣网","event":"shareToDouban","lang":"shareWeb_DouBan"},"baidu_tieba":{"label":"百度贴吧","name":"baidu_tieba","selected":true,"value":true,"sortid":"10","shareid":"baidu_tieba","sharetitle":"分享到百度贴吧","event":"shareToTieba","lang":"shareWeb_TieBa"},"Facebook":{"label":"Facebook","name":"Facebook","selected":true,"value":true,"sortid":"11","shareid":"Facebook","sharetitle":"分享到FaceBook","event":"shareToFacebook","lang":"shareWeb_Facebook"},"Twitter":{"label":"Twitter","name":"Twitter","selected":true,"value":true,"sortid":"12","shareid":"Twitter","sharetitle":"分享到Twitter","event":"shareToTwitter","lang":"shareWeb_Twitter"},"LinkedIn":{"label":"LinkedIn","name":"LinkedIn","selected":true,"value":true,"sortid":"13","shareid":"LinkedIn","sharetitle":"分享到linkedIn","event":"shareToLinkedin","lang":"shareWeb_Linkedin"},"whatsapp":{"label":"whatsapp","name":"whatsapp","selected":true,"value":true,"sortid":"15","shareid":"whatsapp","sharetitle":"分享到whatsapp","event":"shareToWhatsapp","lang":"shareWeb_whatsapp"},"line":{"label":"line","name":"line","selected":true,"value":true,"sortid":"15","shareid":"line","sharetitle":"分享到line","event":"shareToLine","lang":"shareWeb_line"},"qq_weibo":{"label":"腾讯微博","name":"qq_weibo","selected":true,"value":true,"sortid":"3","shareid":"qq_weibo","sharetitle":"分享到腾讯微博","event":"shareToQQwb","lang":"shareWeb_QQWeiBo"},"peopleBlog":{"label":"人民微博","name":"propleBlog","selected":true,"value":true,"sortid":"14","shareid":"propleBlog","sharetitle":"分享到人民微博","event":"shareToPeopleBlog","lang":"shareWeb_peopleBlog"}}