Tasks #2526

EwoutH · 2024-12-03T08:15:54Z

EwoutH
Dec 3, 2024
Maintainer

This is partly implemented in our new Action. See #3304 and #3308.

Introduction

ABM started quite simple, with Agents doing (or not doing) things in a fixed timestep. Then, we evolved it to allow agents to do things at any time, first with the DiscreteEventScheduler and then even better with full support for discrete event scheduling. However, Mesa doesn't really have a concept of a time duration to do something. You can emulate it by scheduling the next task for an agent some duration in advance, but this has several limitation:

You don't know if an Agent is still busy
You can accidentally schedule things while the agent was supposed to (still) be doing something else
It's non-obvious to give the reward for completing something when it's actually finished
There's no good way to cancel or interrupt something an agent is doing
There's no convenient way to let an Agent set priorities

To solve all of this, I would like to propose the Task.

The Task

A Task is an object that represents a specific activity an Agent is performing over a period of time, with a defined duration, priority, and completion state. It encapsulates all the information needed to track what an Agent is doing, when they started, when they'll finish, what happens upon completion, and how it can be interrupted.

In my vision, it has a few handy properties:

Time and progress: A tasks has an explicit duration (distribution) function and status. It can either be scheduled
Reward function: Completing the tasks gives your some reward when it's finished. Optionally, it can also give you a reward on partially finishing a tasks. The can be a function. For example, y=x gives you a linear reward: 50% completion maps to 50% of the reward. But y=x^2 will only give you 25% reward onto completing a tasks 50%.
- This also allows for doing something until a desired threshold is reached, for example sleep until you're 80% rested (scheduling this might be a bit tricky with stochastic reward functions).
Occupation: A tasks occupies an agent, so when it's doing one tasks, it can't be simultaneously doing something else (see future considerations below).
Requirements: To perform a tasks, there might be certain requirements. These can be both from your internal states or the environment. To eat, you might need to be hungry and your might need to be on a cell with grass.
- You can get into traditional discrete-event modelling here if you also allow to claim certain resources, like cells, servers, agents, etc.
Interruptible: A tasks can be interrupted by either the agent itself, other agents or an environmental event or condition.
Resumable: An interrupted tasks can be resumable or not (a boolean). This means you can pause doing it, so something else (of higher priority) in between, and continue where you left of. Internally this can be modelled as it being rescheduled with with a non-zero progress % status, which results in a shorter remaining time when resumed then the full tasks would be.

Examples

Let's talk through a few examples.

Wolf-Sheep

Imagine you are a sheep. Most of the day you're searching for vegetation and eating it. This could be modelled as a relatively long tasks, you might do this for hours a day. However, you're also always spending some attention on spotting wolves. Once you spot one, you break of your food-search tasks, and highly priority you're fleeing tasks.

Ant Colony

Consider ants building a new nest chamber. Individual ants perform a "Dig Soil" task with progress determined by soil hardness. When they've gathered enough soil, they start a "Transport Material" task with duration based on distance and load. However, if they encounter a colony member with a stronger urgent pheromone signal, they can interrupt their current task to assist - perhaps there's a cave-in that requires immediate attention. The reward function for construction tasks is non-linear: partial nest chambers provide little benefit, but completing rooms gives large colony-wide bonuses. This creates emergent cooperation where ants dynamically adjust their tasks based on colony needs and local information.

Medieval Market

Picture merchants in a medieval marketplace. Each has a primary "Sell Goods" task that's continuously active but with variable reward rates based on time of day and location. They can temporarily interrupt selling to perform a "Haggle" task when high-value customers appear - this has a uncertain duration and could result in either high rewards or wasted time. Meanwhile, they're also managing a "Watch for Thieves" task that runs parallel at low efficiency. If they spot suspicious behavior, they can initiate an immediate "Protect Goods" task that prevents theft but halts all trading. The priority of these tasks shifts based on crowd density, reported thefts, and remaining inventory - creating dynamic market behaviors.

Desire-based modelling

One of the things that this might allow is what I call desire-based modelling. Agents have internal desires they want to fulfil, and they have states with how much these desires are fulfilled. These desires can decline based on time, environment or tasks involved. For example, agents might have desire to stay warm, be fed and have sex. The larger the gap between an agent state and their desired state, the higher a task can be prioritized.

(Future) ideas to further develop

Priorities and scheduling

It seems obvious, that once you have tasks, you would like a system to agents to automatically order and schedule them. Some mechanisms to do that would be useful, either based on priorities, urgency, desires and/or expected rewards.

Time sensitives rewards.

Generally there is a benefit of doing something, which diminishes over time. Sometimes this is a hard cut-off, where the full reward is available upon some time and then it springs to 0. Other times it might just diminish over time.

Nestedness

It seems logical that tasks could be nested. There is the big tasks "Feed yourself" which might consists about searching for food and eating it. Searching for food again consists about looking for potential directions and moving to a potential direction.

Permission management

Who can interrupt which tasks? Who can claim which resources? Some mechanisms to let agents fight over that might be nice.

Multiple simultaneous tasks

The most simple implementation here would say an Agent has a capacity to handle X tasks, and if a tasks takes up some part of X the remainder. However, that might also need to include a function that scales the tasks efficiency with the portion (this might not always be linear).

Task States and Transitions

Tasks would benefit from more granular states beyond just active/interrupted/completed, such as "preparing", "waiting", or "paused". This needs clear rules about valid transitions between states and should track transition history for analysis.

Failing

Tasks need robust failure handling - what happens when requirements become false during execution, how many retry attempts are allowed, and how to handle timeouts for stuck tasks. This connects to both state management and multi-agent coordination.

Multi-Agent Coordination

Some tasks require multiple agents working together, raising questions about delegation, sharing, communication and conflict resolution. This could include task trading, bidding systems, and coordinated resource usage.

Learning and Adaptation

Tasks could improve in efficiency with repetition, agents could learn optimal task sequences, and historical performance could affect future decisions. This allows for dynamic adaptation to changing conditions.

Let's discuss

Very curious what everybody thinks about this. Give any thoughts and feedback you like, but I also have a few specific points:

Is this useful in core Mesa? Considering the complexity and such?
What scientific literature exists around this concept?
What kind of use cases would you have for Tasks?
Conceptually, is anything crucial here missing?

quaquel · 2024-12-03T08:51:05Z

quaquel
Dec 3, 2024
Maintainer

This seems closely related to the process interaction formalisms for discrete event simulation. See e.g. Jacobs & Verbraeck (2004).

0 replies

EwoutH · 2024-12-03T09:22:43Z

EwoutH
Dec 3, 2024
Maintainer Author

@projectmesa/maintainers, I would love to hear everybody thoughts!

@harshmahesheka, you spend quite a bit of time on Reinforcement Learning with Mesa. Would you view this as an useful construct to build learning models on?

0 replies

EwoutH · 2024-12-05T20:53:17Z

EwoutH
Dec 5, 2024
Maintainer Author

Tasks turned out to not really be useful on their own. Tasks become really powerful when an Agent can initiate (schedule) them based on it's current states (and maybe based on request from other agents). Therefore, you need a few components:

Internal states that degrade based on time / events
A loss function that determines the negative effect of each degraded state.
Agent decision logic that minimizes the combined loss over all internal states

Ideally you want this modular and extendable, so that you can use RL to tune loss functions and/or decision logic.

Scope creep is hitting hard. Going to spend some more brain cycles on this on how to make it minimal. If anyone knows something that does this already, highly recommended (again @harshmahesheka).

3 replies

EwoutH Dec 5, 2024
Maintainer Author

Existing concepts / projects in this space seem to be:

SPADE (Smart Python Agent Development Environment)
- Python library that implements FIPA agent communication protocols. Has built-in behavior scheduling and finite state machines. Agents can have different behaviors triggered by internal states. However, it's more focused on agent communication than decision-making
PDDL (Planning Domain Definition Language) with Python implementations like pddlpy and pypddl
- These allow defining states, actions, and goals formally. Good for planning sequences of actions. But they're more rigid/formal.
Behavior Trees implementations like py_trees:
- Used heavily in game AI and robotics. Good at handling state-based decision making. Can compose complex behaviors from simple ones. However, they're more focused on immediate decision-making than long-term planning

PettingZoo is also interesting, which is like OpenAI Gym but for multi-agent RL. It has Support for internal agent states, Reward/loss functions, Multi-agent interactions and Compatibility with various RL algorithms.

EwoutH Dec 5, 2024
Maintainer Author

We're hitting a weird spot here that we want Agents to have some autonomy, but we don't want to go full black-box RL (because then the value of ABM disappears).

EwoutH Dec 5, 2024
Maintainer Author

There's a lot more here:

Belief-Desire-Intention (BDI) architecture from AI - agents have beliefs about the world, desires they want to achieve, and intentions (current committed plans)
Subsumption architecture - layered behaviors with priorities
Utility-based agents from Russell & Norvig's AI text
Homeostatic agent architectures where agents try to maintain optimal internal states
PECS (Physical, Emotional, Cognitive, Social) reference model for agent behavior
GOAP (Goal Oriented Action Planning) uses A* pathfinding through possible actions to satisfy goals. Each action has preconditions and effects, goals have associated costs/priorities and is used successfully in game AI.

The challenge is finding something that's:

Lightweight enough to integrate with Mesa
Flexible enough for different types of agent behavior
Has good support for both hand-tuned and learned policies
Can handle both discrete events and continuous state evolution

jackiekazil · 2024-12-13T03:53:07Z

jackiekazil
Dec 13, 2024
Maintainer

While this might be useful to Mesa, I wonder if this would be best external to Mesa to start as a add-on? And it conceptually structures the tasks around the current time construct, where a task takes X unites of time to complete with all sorts of different way to structure.

2 replies

jackiekazil Dec 13, 2024
Maintainer

Dare I say -- the proposal for how this is constructed and the background research might be a paper in itself.

EwoutH Dec 13, 2024
Maintainer Author

Thanks! I think personally the basic blocks could be in Mesa as a module, on which everyone can build.

It amounts to quite a serious task indeed ;)

DashamiJituri · 2026-03-22T14:43:05Z

DashamiJituri
Mar 22, 2026

Hi — I've been reading through this discussion alongside #2538 (Behavioral Framework) while working on a GSoC 2026 proposal around behavioral patterns in Mesa.

The idea of tasks with explicit duration connects directly to something I noticed while extending Wolf–Sheep — actions like reproduction and eating are currently forced to be instantaneous because step() has no concept of duration. I left a more detailed comment with code in #2538 if useful context.

Looking forward to seeing how the Tasks and Behavioral Framework discussions converge.

0 replies

ammonia2 · 2026-03-27T15:15:14Z

ammonia2
Mar 27, 2026

Hi @EwoutH, I'm writing my GSoC proposal centred around Behavioural framework #2538 and Tasks.

While building a SARSA 2v2 football simulation system using Mesa, I encountered some problems mentioned in this discussion: manually engineering a reward function for different states, abstracting tasks like passing into a single timestep without any duration. I used RL (SARSA) to tune decision logic implicitly through Q-values, but the loss function and state degradation were completely manual and tangled into reward shaping. Making those three components explicit and modular would have let me swap the decision logic, for example for example upgrading from SARSA to a neural network policy, without rewriting the reward structure or state encoding., without rewriting the reward structure or state encoding. Right now those are all tangled together.

One thing that is unclear to me is that does a Task own its completion reward or does the State Management layer observe Task completion and fire rewards externally?

4 replies

EwoutH Mar 27, 2026
Maintainer Author

Could you maybe give a simple, minimal example of this? How you needed to do it with current Mesa and how you propose it should work?

ammonia2 Mar 27, 2026

Okay, here's what I did:
Currently in my football model, state, decision logic, and reward are all fused in the agent's step():

def step(self):
    state = (rowBin, colBin, ballDist, hasBall, oppNearby)
    action = self.choose_action(state)  # SARSA Q-table
    
    if action == "pass":
        # no duration, instantaneous
        self.pass_ball()
        reward = 0.3 if self.closer_to_goal() else -0.1
    
    self.update_q(state, action, reward, next_state)

The problems: passing has no duration, reward logic is hardcoded inside step(), and swapping SARSA for a different policy means rewriting all of this.

With Tasks and explicit State Management it would look like:

class PassTask(Task):
    duration = 2  # takes 2 ticks
    def on_complete(self, agent):
        # Task completes, State layer observes and fires reward
        pass

class FootballAgent(Agent):
    def step(self):
        if self.state["has_ball"]:
            self.initiate(PassTask())  # decision logic separate

This is also where my question narrows down. Should on_complete fire the reward directly, or should a separate State layer observe task completion and handle it?

EwoutH Mar 27, 2026
Maintainer Author

passing has no duration

Would our new Action class help there?

ammonia2 Mar 27, 2026

Yes, my bad for overlooking that experimental feature.

For the other two problems, I think the existing signals system (Observable, observe()/emit()) is actually the right foundation rather than inventing new abstractions:

Reward logic hardcoded in step():
Instead of a custom RewardFunction class, on_complete can emit a signal and the RL policy subscribes to it externally:

class footballAgent(Agent):
    lastAction: str = Observable()  # fires CHANGED when assigned

class passAction(Action):
    def on_complete(self):
        self.agent.lastAction = "pass"  # this triggers the signal automatically

# external reward handling
agent.observe("lastAction", ObservableSignals.CHANGED, rewardHandler)

def rewardHandler(agent, signal):
    agent.rewardBuffer += 0.3 if agent.closerToGoal() else -0.1

This keeps reward logic fully separate.

Decision logic tangled in step():
A thin Policy layer that maps state to an Action subclass, sitting on top of Action:

class sarsaPolicy:
    def select(self, agent):
        state = agent.getState()
        actionClass = self.qTable.bestAction(state)
        return actionClass(agent)

class footballAgent(Agent):
    def step(self):
        self.startAction(self.policy.select(self))

Swapping to a neural policy is just self.policy = neuralPolicy().

I want to ask is the intention that reward orchestration hooks into the existing signals system like this? And is a Policy/decision layer something the behavioral framework plans to formalise?

samuelmohel · 2026-03-27T21:03:27Z

samuelmohel
Mar 27, 2026

I've been exploring this discussion as part of a GSoC 2026 proposal for the Behavioral Framework project. I implemented the Task concept from this thread as a working proof-of-concept, a model-agnostic Task and TaskQueue class with duration, priority, interruptible/resumable flags, and pluggable reward functions, demonstrated via an ant colony model where cave-in pheromone signals interrupt ongoing dig tasks in real time. PR #445 on mesa-examples: mesa/mesa-examples#445. One design question: I used integer step-counts for task duration. Is there a preferred approach for integrating this with DiscreteEventScheduler's float time, or should Task accept both?

0 replies

DashamiJituri · 2026-03-28T16:55:37Z

DashamiJituri
Mar 28, 2026

Following up on my earlier comment — after reading @ammonia2's signal-based approach and @samuelmohel's PR #445 implementation, I think the cleanest integration for needs-based agents would be:
A DriveManager that subscribes to Task.COMPLETED via Mesa's existing signals system, rather than having the Task own the drive update directly. This keeps Task agnostic about what drives exist, and lets you swap the drive model without touching Task logic.
From PR #432: when a wolf's FeedTask completes, hunger resets to 0.0 — currently that's hardcoded in the agent. With signals it becomes:
agent.observe("active_task", ObservableSignals.CHANGED, drive_manager.on_task_complete)
Is this the direction the behavioral framework intends, or is drive management expected to live inside Task itself?

0 replies

Nuwanmnd · 2026-03-30T08:13:14Z

Nuwanmnd
Mar 30, 2026

Hi @EwoutH and @quaquel
I’m planning to apply for GSoC for the Behavioral Framework project.

I’ve started exploring Mesa locally and forked the GSoC learning-space repo. My current plan is to build two small models there: a needs-based human agent model and a predator-prey model with internal drives such as hunger and fear. My goal is to learn Mesa through implementation and use that experience to shape a stronger proposal.

I wanted to ask:

Does this sound like a good starting direction for the Behavioral Framework project?
Between needs-based, task-oriented, and learning-based behavior, which would be most valuable to emphasize early in a proposal?

Thanks! I’d really appreciate any guidance.

2 replies

EwoutH Mar 30, 2026
Maintainer Author

To be honest, with the deadline being tomorrow, I don't think you will have enough time to dive into this complex topic before GSoC.

If you're also interested in contributing outside GSoC we're happy to help and discuss this further.

Nuwanmnd Mar 30, 2026

Thanks for the honest reply. I really appreciate it.
That makes sense. I started too late for a strong GSoC application on this topic.
I’m still genuinely interested in the Behavioral Framework direction outside GSoC, though, and I’d like to keep exploring it through a small learning-space model or two so I can understand Mesa better from a modeller’s perspective.

Uh oh!

Tasks #2526

Uh oh!

Uh oh!

EwoutH Dec 3, 2024 Maintainer

Introduction

The Task

Examples

Wolf-Sheep

Ant Colony

Medieval Market

Desire-based modelling

(Future) ideas to further develop

Priorities and scheduling

Time sensitives rewards.

Nestedness

Permission management

Multiple simultaneous tasks

Task States and Transitions

Failing

Multi-Agent Coordination

Learning and Adaptation

Let's discuss

Replies: 9 comments · 11 replies

Uh oh!

quaquel Dec 3, 2024 Maintainer

Uh oh!

EwoutH Dec 3, 2024 Maintainer Author

Uh oh!

EwoutH Dec 5, 2024 Maintainer Author

Uh oh!

EwoutH Dec 5, 2024 Maintainer Author

Uh oh!

EwoutH Dec 5, 2024 Maintainer Author

Uh oh!

EwoutH Dec 5, 2024 Maintainer Author

Uh oh!

jackiekazil Dec 13, 2024 Maintainer

Uh oh!

jackiekazil Dec 13, 2024 Maintainer

Uh oh!

EwoutH Dec 13, 2024 Maintainer Author

Uh oh!

Uh oh!

Uh oh!

Uh oh!

EwoutH Mar 27, 2026 Maintainer Author

Uh oh!

Uh oh!

EwoutH Mar 27, 2026 Maintainer Author

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

EwoutH Mar 30, 2026 Maintainer Author

Uh oh!

EwoutH
Dec 3, 2024
Maintainer

Replies: 9 comments 11 replies

quaquel
Dec 3, 2024
Maintainer

EwoutH
Dec 3, 2024
Maintainer Author

EwoutH
Dec 5, 2024
Maintainer Author

EwoutH Dec 5, 2024
Maintainer Author

EwoutH Dec 5, 2024
Maintainer Author

EwoutH Dec 5, 2024
Maintainer Author

jackiekazil
Dec 13, 2024
Maintainer

jackiekazil Dec 13, 2024
Maintainer

EwoutH Dec 13, 2024
Maintainer Author

EwoutH Mar 27, 2026
Maintainer Author

EwoutH Mar 27, 2026
Maintainer Author

EwoutH Mar 30, 2026
Maintainer Author