Datasets:

OpenDFM
/

MobA-MobBench

task_id int64 1 50	task_type stringclasses 5 values	task_zh stringlengths 8 58	task_en stringlengths 23 201	applications sequencelengths 1 2	preparation_zh stringlengths 0 27	preparation_en stringlengths 0 109	total_milestone_scores int64 1 6	scoring_milestones_zh sequencelengths 1 6	scoring_milestones_en sequencelengths 1 6	human_expert_steps sequencelengths 3 3
1	easy	在12306中进入“我的”页面，在常用功能中将app从标准模式切换到敬老版。	Go to the 'My' page in the 12306 app, and switch the app from standard mode to senior mode under common functions.	[ "12306 (China Railway)" ]	切换到标准模式	switch to standard mode	1	[ "完成任务" ]	[ "task completion" ]	[ 5, 5, 8 ]
2	easy	在B站搜索“高等数学”并播放第一个视频。	Search 'Advanced Mathematics' in Bilibili and play the first video.	[ "BiliBili" ]			1	[ "完成任务" ]	[ "task completion" ]	[ 5, 6, 5 ]
3	easy	使用相机的夜景模式拍一张照片。	Take a photo using the camera's night mode.	[ "Camera" ]			1	[ "完成任务" ]	[ "task completion" ]	[ 3, 4, 3 ]
4	easy	关闭最早的一个打开的闹钟。	Turn off the earliest enabled alarm.	[ "Clock" ]	设置并打开数个闹钟	Set and turn on several alarms.	1	[ "完成任务" ]	[ "task completion" ]	[ 2, 2, 2 ]
5	easy	查询《人工智能》这部电影的豆瓣评分。	Check the Douban rating of the movie 'Artificial Intelligence'.	[ "Douban" ]			1	[ "完成任务" ]	[ "task completion" ]	[ 5, 4, 5 ]
6	easy	在谷歌日历上创建一个新任务，标题为“论文写作”。	Create a new task on Google Calendar titled 'Writing Paper'.	[ "Google Calendar" ]			1	[ "完成任务" ]	[ "task completion" ]	[ 5, 5, 5 ]
7	easy	帮我查询我的华住会会员情况，并告诉我最晚可以几点退房。	Help me check my H World membership and tell me the latest checkout time.	[ "H World" ]	登录账号	logging in an account	1	[ "完成任务" ]	[ "task completion" ]	[ 3, 2, 3 ]
8	easy	在京东搜索麦克风，并将第一个商品添加到购物车。	Search for microphones on JD and add the first product to the cart.	[ "JD" ]			1	[ "完成任务" ]	[ "task completion" ]	[ 6, 5, 6 ]
9	easy	将麦当劳app的语言切换为英文。	Switch the language of the McDonald's app to English.	[ "McDonald's" ]	切换语言为中文	switch the language to Chinese	1	[ "完成任务" ]	[ "task completion" ]	[ 7, 6, 7 ]
10	easy	查看明天的天气。	Check tomorrow's weather.	[ "Weather" ]			1	[ "完成任务" ]	[ "task completion" ]	[ 2, 1, 1 ]
11	medium	在时刻表中查询明天从上海到北京的编号为G104的列车预计几点钟到南京。	Check the timetable for train G104 from Shanghai to Beijing tomorrow, and find out what time it is expected to arrive in Nanjing.	[ "12306 (China Railway)" ]			3	[ "进入时刻表查询界面", "车次正确", "完成任务" ]	[ "enter the timetable page", "enter train number correctly", "task completion" ]	[ 13, 13, 9 ]
12	medium	从B站“热门”页面进入排行榜，播放排名第一的视频，关注这个账号并添加这个视频到收藏夹中。	Go to the leaderboard from the 'Hot' page of BiliBili, play the top-ranked video, follow the account, and add the video to the favorites.	[ "BiliBili" ]	取消关注这个目标账号，关闭自动连播功能	unfollow this target account, turn off the auto-play function	4	[ "进入排行榜", "播放视频", "关注账号", "添加收藏" ]	[ "enter the leaderboard page", "play video correctly", "follow the account", "add to favorites" ]	[ 6, 6, 6 ]
13	medium	设置最长延迟的定时自拍。	Set the timer selfie with the longest delay.	[ "Camera" ]	切换到后置摄像头	switch to the back camera	2	[ "设置延迟", "切换前置" ]	[ "set delay", "switch to the front camera" ]	[ 4, 4, 6 ]
14	medium	在世界时间列表中添加香港。	Add Hong Kong in list of world clock.	[ "Clock" ]	从世界时间列表中移除香港	remove Hong Kong from the list of world clock	2	[ "进入添加界面", "完成任务" ]	[ "enter the page to add a city", "task completion" ]	[ 6, 6, 6 ]
15	medium	搜索并进入“豆瓣电影Top250”榜单，给排名第二的电影最顶部的短评点赞。	Search 'Douban Movie Top 250' leaderboard and like the top comment of the second-ranked movie.	[ "Douban" ]	这条短评没有被点赞	this comment has not been liked	2	[ "进入榜单", "完成任务" ]	[ "enter the leaderboard page", "task completion" ]	[ 11, 5, 8 ]
16	medium	在谷歌日历上创建一个新日程，日期为明天，时间为全天，标题为“公司年会”。	Create a new all-day event on Google Calendar for tomorrow titled 'Company Annual Meeting'.	[ "Google Calendar" ]			2	[ "日期时间正确", "完成任务" ]	[ "correct date and time", "task completion" ]	[ 8, 6, 6 ]
17	medium	在华住会app上查询评分最高的汉庭酒店的起价。	Use the H World app, search for the starting price of the nearest Hanting Hotel to me.	[ "H World" ]			2	[ "使用评分最好排序", "完成任务" ]	[ "sort by highest rating", "task completion" ]	[ 7, 5, 8 ]
18	medium	寻找京东商城中最近待评价的一个商品，给出满分好评，并写一段评语，添加两张照片。	Find the most recent product awaiting comment on JD, give it a full five-star rating, write a comment, and add two photos.	[ "JD" ]	有未评价的订单	there are orders that have not been commented on	2	[ "进入评价界面", "完成任务" ]	[ "enter comment page", "task completion" ]	[ 14, 8, 12 ]
19	medium	给最近的一笔麦当劳订单开发票，并选择“上海交通大学”抬头。	Invoice the most recent McDonald's order, and select 'SJTU' for invoice title.	[ "McDonald's" ]	有未开发票订单	there is an order and no invoice has been issued	3	[ "进入订单界面", "选取抬头正确", "完成任务" ]	[ "enter order list page", "select invoice title correctly", "task completion" ]	[ 9, 9, 12 ]
20	medium	查看今晚日落的时间。	Check the sunset time of tonight.	[ "Weather" ]			1	[ "完成任务" ]	[ "task completion" ]	[ 4, 2, 3 ]
21	hard	购买一张明天从上海到北京，出发时间在九点到十一点之间的高铁票。	Buy a high-speed train ticket for tomorrow from Shanghai to Beijing, with departure time between 9 AM and 11 AM.	[ "12306 (China Railway)" ]	出发站和到达站与首页展示的不同，且有直达余票即可	departure and arrival stations differ from those on the homepage, and direct tickets must be available.	5	[ "日期正确", "车站正确", "车次选择合适", "添加乘车人", "完成任务" ]	[ "correct date", "correct station", "suitable train selection", "add passenger", "complete task" ]	[ 20, 20, 18 ]
22	hard	在B站搜索并关注“X-LANCE实验室”账号，播放最新的视频，并撰写一段友善的评论。	Search in BiliBili and follow the 'X-LANCE' Lab account, play the newest video and write a friendly comment.	[ "BiliBili" ]	取消关注这个账号，关闭自动连播功能	unfollow this account, turn off the auto-play function	4	[ "进入账号主页", "关注账号", "播放视频", "撰写评论" ]	[ "enter homepage of this account", "follow this account", "play video correctly", "write comment" ]	[ 12, 10, 10 ]
23	hard	将相机切换到视频模式，打开补光灯，画质设置成“FHD 30FPS”，录制时间超过二十秒以后停止录制。	Switch the camera to video mode, turn on the fill light, set the video quality to 'FHD 30FPS', and stop recording after more than twenty seconds.	[ "Camera" ]	切换到非视频模式，重置所有设置	switch to non-video mode, reset all settings	3	[ "打开补光灯", "切换画质", "完成20s录制" ]	[ "turn on the fill light", "switch video quality", "complete 20s recording" ]	[ 9, 9, 8 ]
24	hard	创建一个十点的闹钟，标题为“上班”，在每周一至四使用振动提醒。	Create an alarm at 10 o'clock titled 'Work' with vibration reminders on every Monday to Thursday.	[ "Clock" ]			5	[ "时间正确", "标题正确", "周期重复正确", "关闭响铃打开振动", "完成任务" ]	[ "correct time", "correct title", "correct period repetition", "turn off the ringtone and turn on the vibration", "task completion" ]	[ 14, 45, 15 ]
25	hard	在豆瓣中搜索电影《肖申克的救赎》，标记为“看过”，给五颗星的评分，并留下一条称赞的影评。	Search for the movie 'The Shawshank Redemption' on Douban, mark it as 'watched', rate it with five stars, and leave a positive review.	[ "Douban" ]	删除这部电影的之前的标记、评分和影评	Remove the previous mark, rating, and review of this movie.	4	[ "进入电影详情界面", "成功标记", "成功打分", "留下影评" ]	[ "enter the movie details page", "successfully marked", "successfully rated", "leave a review" ]	[ 10, 10, 9 ]
26	hard	在谷歌日历上创建一个周期日程，标题为“计算机视觉课程”，时间为每周三晚上六点到八点，从本周开始，重复八次。	Create a recurring event on Google Calendar titled 'Computer Vision Course' scheduled for every Wednesday from 6 PM to 8 PM, starting this week and repeating eight times.	[ "Google Calendar" ]			3	[ "时间标题正确", "周期重复正确", "完成任务" ]	[ "correct time and title", "correct period repetition", "task completion" ]	[ 20, 52, 21 ]
27	hard	在华住会app上帮我订一个成都的酒店，在天府广场附近，今天入住后天离店，选择价格最低排序的第二个酒店，使用微信支付。	Book a hotel in Chengdu on the H World app, near Tianfu Square, check-in today and check-out the day after tomorrow. Choose the second cheapest hotel listed when sorted by price, and use WeChat to pay.	[ "H World" ]			6	[ "日期正确", "城市地点正确", "价格正确", "进入预订界面", "提交订单", "付款方式正确" ]	[ "correct dates", "correct city and hotel location", "correct price", "enter booking page", "submit order", "correct payment method" ]	[ 17, 16, 15 ]
28	hard	在京东搜索“思必驰办公设备旗舰店”，查询并购买其店铺中最便宜的麦克风，选择浅色款式并购买两份。	Search for AISpeech store on JD, find and buy the cheapest microphone in the store, choose the light-colored style, and buy two.	[ "JD" ]			3	[ "进入店铺主页", "商品选择正确", "完成任务" ]	[ "enter the store homepage", "correct product selection", "task completion" ]	[ 10, 12, 12 ]
29	hard	点一份立即到店取餐的“随心配1+1”超值套餐，选择麦香鸡和迷你朱古力新地，加入购物车，下单并选择支付宝支付。	Order a '1+1 Mix & Match' for immediate in-store pickup, choose McChicken and Mini Chocolate Sundae, add to cart, place the order, and choose Alipay to pay.	[ "McDonald's" ]			5	[ "找到套餐", "选择餐品", "结算订单", "付款方式正确", "完成任务" ]	[ "find the '1+1' meal", "select the meal", "place the order", "correct payment method", "task completion" ]	[ 15, 13, 15 ]
30	hard	在天气应用中添加“成都”这个城市，并告诉我明天的温度区间。	Add 'Chengdu' in the weather app and tell me the temperature range for tomorrow.	[ "Weather" ]	城市列表中没有成都	Ensure that Chengdu is not listed in the city list.	3	[ "进入城市列表", "成功添加", "信息正确" ]	[ "enter the city list", "successfully added", "correct information" ]	[ 9, 4, 5 ]
31	indirect	帮我订一张后天的高铁票，18:00之前回北京。	Book a high-speed train ticket for the day after tomorrow, returning to Beijing before 6:00 PM.	[ "12306 (China Railway)" ]	需要使用当前定位作为出发站	should use the current location as the departure station	5	[ "日期正确", "车站正确", "车次选择合适", "添加乘车人", "完成任务" ]	[ "correct date", "correct stations", "suitable train selection", "add passenger", "task completion" ]	[ 20, 20, 18 ]
32	indirect	没流量了，现在手机上有什么能看的视频？	I'm out of mobile data, what videos can I still watch on the phone?	[ "BiliBili" ]	提前缓存多个视频，关闭网络连接，可以使用其他视频应用	download multiple videos in advance, turn off the network connection, it's ok to use other video applications	2	[ "打开视频应用", "提取缓存视频信息" ]	[ "open video application", "extract downloaded video information" ]	[ 4, 3, 3 ]
33	indirect	把刚刚拍的最后一张照片删掉。	Delete the last photo taken recently.	[ "Camera" ]	拍几张不同的照片，并在其删除多张情况发生时手动停止任务	take several different photos, and stop the task manually when multiple deletions occur	2	[ "删除照片", "删除一张照片之后结束任务" ]	[ "delete photo", "end task after deleting one photo" ]	[ 4, 5, 4 ]
34	indirect	两个小时后提醒我。	Remind me in two hours.	[ "Clock" ]			2	[ "进入计时器界面", "完成任务" ]	[ "enter the timer page", "task completion" ]	[ 7, 5, 6 ]
35	indirect	帮我看看电影院即将上映的电影中最热门的是哪个？	Help me check which is the most popular movie among the upcoming releases in theaters.	[ "Douban" ]			2	[ "打开豆瓣", "完成任务" ]	[ "open Douban", "task completion" ]	[ 6, 3, 4 ]
36	indirect	记录我明天上午十点到十二点在上海交通大学闵行校区上的“自然语言处理”课程。	Record that I have a 'Natural Language Processing' class from 10 AM to 12 PM tomorrow at Shanghai Jiao Tong University, Minhang Campus.	[ "Google Calendar" ]			4	[ "进入创建日程界面", "时间正确", "地点正确", "完成任务" ]	[ "enter the calendar creation page", "correct time", "correct location", "task completion" ]	[ 16, 40, 17 ]
37	indirect	告诉我附近可以入住的最近酒店，并帮我订一个。	Tell me about the nearest available hotel nearby and book it for me.	[ "H World" ]			4	[ "选择当前定位", "酒店选择正确", "提交订单", "完成任务" ]	[ "select current location", "correct hotel selection", "submit order", "task completion" ]	[ 10, 10, 10 ]
38	indirect	帮我问一下客服，购物车最上面的那个商品有没有优惠。	Help me ask customer service if there is any discount for the top item in the shopping cart.	[ "JD" ]	提前将商品加入购物车	add the item to the shopping cart in advance	3	[ "选择正确商品", "进入客服页面", "完成任务" ]	[ "select the correct item", "enter customer service page", "task completion" ]	[ 7, 8, 8 ]
39	indirect	点一份“麦辣鸡腿堡套餐”堂食	Order a McSpicy Chicken Filet Burger Combo for dine-in.	[ "McDonald's" ]			3	[ "找到套餐", "结算订单", "完成任务" ]	[ "find the combo", "place the order", "task completion" ]	[ 13, 13, 13 ]
40	indirect	明天我需要带雨伞吗？	Do I need to bring an umbrella tomorrow?	[ "Weather" ]			1	[ "完成任务" ]	[ "task completion" ]	[ 3, 1, 2 ]
41	cross_app	帮我把最近的一个火车行程添加到日程，标题格式为“{出发站}——{车次}——{到达站}”，并设置具体时间范围。	Help me add my latest train journey to the schedule, titled in format of '{Departure Station}——{Train Number}——{Arrival Station}', and set exact time range.	[ "12306 (China Railway)", "Google Calendar" ]			3	[ "进入车票列表", "标题正确", "时间正确" ]	[ "enter ticket list page", "correct title", "correct time" ]	[ 23, 10, 12 ]
42	cross_app	在B站播放“豆瓣电影Top250”排名第三的那部电影。	Play the movie ranked third in the 'Douban Movie Top 250' on BiliBili.	[ "BiliBili", "Douban" ]			2	[ "电影正确", "成功播放" ]	[ "correct movie", "play successfully" ]	[ 12, 13, 12 ]
43	cross_app	打开相机并告诉我镜头前的物品是什么，在京东搜索这类商品，并选择较为相似的一项加入购物车。	Open the camera and tell me what item is in front of the lens, search for this type of product on JD, and select a similar item to add to the shopping cart.	[ "Camera", "JD" ]	后置摄像头对着一个常见物品	point the back camera at a common object	3	[ "回答正确", "成功搜索", "加入购物车" ]	[ "correct answer", "search successfully", "add to shopping cart" ]	[ 10, 8, 8 ]
44	cross_app	使用长焦镜头拍一张照片，并分享到微信朋友圈，写一段话，并在最后加上注释“这是由MobA手机助手自动发布的朋友圈”。	Take a photo with a telephoto lens and share it on WeChat Moments, write a few words with the note 'This is an automatically posted Moments by MobA Mobile phone Assistant' in the end.	[ "Camera", "WeChat" ]	提前登录微信，镜头对着远处的物体	log in to WeChat in advance, point the camera at a distant object	4	[ "拍摄照片", "从图库进入分享菜单", "文案正确", "成功发送朋友圈" ]	[ "take a photo", "enter share menu from gallery", "correct text content", "successfully send moments" ]	[ 10, 10, 10 ]
45	cross_app	我明天几点开会？定一个提前两个小时的闹钟提醒我。	What time is my meeting tomorrow? Set an alarm for two hours before the meeting to remind me.	[ "Google Calendar", "Clock" ]	设置一个明天早上十点开始的会议日程	set a meeting schedule that starts at 10 AM tomorrow morning	2	[ "回答正确", "成功添加闹钟" ]	[ "correct answer", "add alarm successfully" ]	[ 10, 11, 12 ]
46	cross_app	我在日历中记录了我的日程安排，现在请你告诉我去深圳旅游那天的天气情况。	Check my calendar for the schedule of my trip to Shenzhen and tell me the weather forecast for that day.	[ "Google Calendar", "Weather" ]	在日历中新建一个后天的全天日程，命名为“深圳之旅”	create a new all-day schedule for the day after tomorrow in the calendar, titled 'Shenzhen Tour'.	4	[ "进入谷歌日历", "成功获取日期", "进入天气", "成功查询天气" ]	[ "enter google calendar", "correct date retrieval", "enter weather", "successful weather query" ]	[ 9, 6, 6 ]
47	cross_app	查询我最近的一笔华住会订单地址，并在B站搜索一下那个城市的旅游攻略。	Check the address of my most recent order in H World and search for travel guides for that city on BiliBili.	[ "H World", "BiliBili" ]	有一个订单，可以是未付款或取消付款状态	there is an order, which can be in unpaid or canceled payment status	3	[ "获取订单", "进入BiliBili", "成功播放视频" ]	[ "retrieve order", "enter BiliBili", "play video successfully" ]	[ 10, 10, 10 ]
48	cross_app	把最近的一笔京东订单的商品链接分享给微信好友，并写一段推荐语。	Share the product link of the most recent order of JD with my WeChat friends, and write a recommendation message.	[ "JD", "WeChat" ]	有一个订单，可以是未付款或取消付款状态	there is an order, which can be in unpaid or canceled payment status	4	[ "进入订单界面", "订单正确", "推荐语合适", "成功发送" ]	[ "enter order list page", "correct order", "suitable recommendation message", "share successfully" ]	[ 12, 9, 10 ]
49	cross_app	帮我在麦当劳点一个麦辣鸡翅，如果现在天气不适合出门就点外卖，不然就堂食	Order a dine-in Spicy McWings at McDonald's. If the current weather is not suitable for going out, order delivery instead.	[ "McDonald's", "Weather" ]			4	[ "查询天气", "选择餐品", "结算订单", "取餐方式正确" ]	[ "check weather", "select meal", "place the order", "correct dining option" ]	[ 15, 15, 14 ]
50	cross_app	如果明天下雨，就定个早上十点的闹钟叫我起床，不然就定八点的闹钟。	If it rains tomorrow, set an alarm for 10 AM to wake me up; otherwise, set the alarm for 8 AM.	[ "Weather", "Clock" ]			2	[ "成功判断明天天气", "成功设置闹钟" ]	[ "correct weather prediction for tomorrow", "set alarm successfully" ]	[ 9, 8, 9 ]

🎮 MobA manipulates mobile phones just like how you would.

🌐 Website | 📃 Paper | 🤗 MobBench | 🗃️ Code

简体中文 | English

🔥 News

[2024.10.18] We open-source MobA on GitHub, and our paper is available on arXiv.

📖 Introduction

Current mobile assistants are limited by dependence on system APIs or struggle with complex user instructions and diverse interfaces due to restricted comprehension and decision-making abilities. To address these challenges, we propose MobA, a novel Mobile phone Agent powered by multimodal large language models that enhances comprehension and planning capabilities through a sophisticated two-level agent architecture. The high-level Global Agent (GA) is responsible for understanding user commands, tracking history memories, and planning tasks. The low-level Local Agent (LA) predicts detailed actions in the form of function calls, guided by sub-tasks and memory from the GA. Integrating a Reflection Module allows for efficient task completion and enables the system to handle previously unseen complex tasks. MobA demonstrates significant improvements in task execution efficiency and completion rate in real-life evaluations, underscoring the potential of MLLM-empowered mobile assistants.

🔧 Deployment

MobA is still under development, and we are keeping updating the code. Please stay tuned!

System Requirements

Make sure you have installed Android Debug Bridge (ADB), and you have connected your Android device to your computer. You should be able to see your devides with command adb devices.

Environment Setup

conda create -n moba python=3.12
conda activate moba
pip install numpy opencv-python openai generativeai pillow colorama

You may also use requirements.txt to install the required packages (However it is not recommended since there are many unused packages).

Run MobA

You need to specify the configuration file in config.yaml before running MobA. You can find the configuration file in the moba folder.

vim ./moba/config.yaml
cd ./moba/agent
python executor.py

You should be able to run MobA smoothly on Windows now. You can find MobBench, the fifty tasks we tested in the paper, on huggingface.

📑 Citation

If you find our work useful, please cite us!

@misc{zhu2024moba,
      title={MobA: A Two-Level Agent System for Efficient Mobile Task Automation}, 
      author={Zichen Zhu and Hao Tang and Yansi Li and Kunyao Lan and Yixuan Jiang and Hao Zhou and Yixiao Wang and Situo Zhang and Liangtai Sun and Lu Chen and Kai Yu},
      year={2024},
      eprint={2410.13757},
      archivePrefix={arXiv},
      primaryClass={cs.MA},
      url={https://arxiv.org/abs/2410.13757}, 
}

📧 Contact Us

If you have any questions, please feel free to contact me via email [email protected].

Downloads last month: 76

Size of downloaded dataset files:

42.8 kB

Size of the auto-converted Parquet files:

20 kB

Number of rows:

Paper for OpenDFM/MobA-MobBench

MobA: A Two-Level Agent System for Efficient Mobile Task Automation

Paper • 2410.13757 • Published Oct 17, 2024 • 32