I believe that using SWE-bench data as training data and then using it for evaluation can cause data leakage issues. For instance, if you look at the golden patch for 'django__django-14730' and Genie's patch, the warning content is exactly the same.

golden patch:

diff --git a/django/db/models/fields/related.py b/django/db/models/fields/related.py
--- a/django/db/models/fields/related.py
+++ b/django/db/models/fields/related.py
@@ -1258,6 +1258,16 @@ def _check_ignored_options(self, **kwargs):
                 )
             )
 
+        if self.remote_field.symmetrical and self._related_name:
+            warnings.append(
+                checks.Warning(
+                    'related_name has no effect on ManyToManyField '
+                    'with a symmetrical relationship, e.g. to "self".',
+                    obj=self,
+                    id='fields.W345',
+                )
+            )
+
         return warnings
 
     def _check_relationship_model(self, from_model=None, **kwargs):

Genie's patch:

diff --git a/django/db/models/fields/related.py b/django/db/models/fields/related.py
--- a/django/db/models/fields/related.py
+++ b/django/db/models/fields/related.py
@@ -1214,6 +1214,7 @@ class ManyToManyField(RelatedField):
             *self._check_relationship_model(**kwargs),
             *self._check_ignored_options(**kwargs),
             *self._check_table_uniqueness(**kwargs),
+            *self._check_symmetrical_related_name(**kwargs),
         ]
 
     def _check_unique(self, **kwargs):
@@ -1260,6 +1261,18 @@ class ManyToManyField(RelatedField):
 
         return warnings
 
+    def _check_symmetrical_related_name(self, **kwargs):
+        if self.remote_field.symmetrical and self.remote_field.related_name:
+            return [
+                checks.Warning(
+                    'related_name has no effect on ManyToManyField with '
+                    'a symmetrical relationship, e.g. to "self".',
+                    obj=self,
+                    id='fields.W345',
+                )
+            ]
+        return []
+
     def _check_relationship_model(self, from_model=None, **kwargs):
         if hasattr(self.remote_field.through, '_meta'):
             qualified_model_name = "%s.%s" % (
       ```

We did check our dataset ahead of time for contamination and made sure that there wasn't any - in fact we didn't train on any Django code (happy to provide evidence of this if necessary), of course we can't do anything about the base model's memory of patches/anything in the SWE set.

Edit: After some further investigation, the file the model edited contains plenty of examples of existing warning messages, for example, this code is directly above the hunk that the model inserted:

if (self.remote_field.limit_choices_to and self.remote_field.through and
                not self.remote_field.through._meta.auto_created):
            warnings.append(
                checks.Warning(
                    'limit_choices_to has no effect on ManyToManyField '
                    'with a through model.',
                    obj=self,
                    id='fields.W343',
                )
            )

        return warnings

Therefore the model saw how the previous warning messages were written and written the new one in the same style, particularly as the existing messages are all written in a very predictable and consistent format.

So why the model even know attaching an example, "self", at the end of this warning message? Such example is not present in the sample warning messages provided.

There is an example self at the end of the warning message I cited in my previous comment, which was already in the file - in fact there are multiple existing warning examples in that file that have the same format.

Sorry for the misunderstanding. But I have carefully reviewed the instance django__django-14730. Before the god_patch and test_patch were applied, it seems there was no warning form with the 'e.g. to xxxx' format. My question is: why, and how did your model learn to output a warning message in that specific format?

I see what you mean, there are multiple (three in total) usages of the phraseology to self in the file at the base commit state for example here, and I should clarify that the entire file was provided to the model during the generation of the patch so all of these usages would have been visible to the model during inference:

    def contribute_to_class(self, cls, name, **kwargs):
        # To support multiple relations to self, it's useful to have a non-None
        # related name on symmetrical relations for internal reasons. The

And if we dig a little deeper into what these examples are referring to when they are talking about 'to self', they are talking about relationships, which is incidentally what the model patch is referring to. I think it's reasonable to assume that as the model is performing inference and is writing the error message, the attention mechanism will obviously pay attention to existing error messages as examples, but also given the links of the model patch to the concept of relationships it makes sense that it would probably be paying attention to areas that are tied to relationships in the file, and all of the usages of to self lie in these ranges.

Now it can't be said with certainty why the model said what it said, but I did store logprobs for every message the model returned during inference, so I've taken a look at the logprobs to try to get a better picture of what was said:

...,
{
            "token": " to",
            "logprob": -0.00051931274,
            "bytes": [
              32,
              116,
              111
            ],
            "top_logprobs": []
          },
          {
            "token": " \"",
            "logprob": -0.00020759332,
            "bytes": [
              32,
              34
            ],
            "top_logprobs": []
          },
          {
            "token": "self",
            "logprob": -0.000003888926,
            "bytes": [
              115,
              101,
              108,
              102
            ],
            "top_logprobs": []
          },
          {
            "token": "\".",
            "logprob": -0.00011844365,
            "bytes": [
              34,
              46
            ],
            "top_logprobs": []
          },
...

Now we can see that the model's certainty of saying the token self in this case is multiple times more certain than any of the surrounding tokens, I have only ever seen certainties this high when the model is repeating something it's already seen in the context window verbatim, rather than generating truly new tokens. Again, truly debugging why the model did what it did is impossible but I think these factors paint a pretty clear picture.

Are there other strings in that file that resemble: 'with a symmetrical relationship, e.g. to "self".'? Or in other files that the agent had seen at that point?

@ofirpress Yes there are, one example is here:

                )

        if symmetrical is None:
            symmetrical = (to == RECURSIVE_RELATIONSHIP_CONSTANT)

        if through is not None and db_table is not None:
            raise ValueError(
                'Cannot specify a db_table if an intermediary model is used.'
            )

        kwargs['rel'] = self.rel_class(
            self, to,
            related_name=related_name,
            related_query_name=related_query_name,
            limit_choices_to=limit_choices_to,
            symmetrical=symmetrical,
            through=through,
            through_fields=through_fields,
            db_constraint=db_constraint,
        )
        self.has_null_arg = 'null' in kwargs

You can see there are plenty of references to symmetrical relationships and calls to self and rel_class etc., notably this snippet is only ~60 lines away from the hunk that the model actually wrote, which I'd call nearby, but that can of course be debated! In the grand scheme of things of a 1700 line file I'd say it's close.

As a final point, it's fairly clear to me that the base model does remember this code from its pretraining, something we can't help unfortunately – as an example I took a chunk of the file from the base commit from the beginning of the file until the first line of the hunk the model wrote: def _check_symmetrical_related_name(self, **kwargs): and gave it gpt-4o and asked it to simply continue writing code from there - the model writes extremely similar code to what's seen in the patch (it varies from run to run but you can see the gist is there), and even correctly defines the name of the function that comes after in the file: def _check_relationship_model(self, from_model=None, **kwargs): – you can try this for yourself here in the playground. I think between this underlying knowledge, and the surrounding context in the file, it's pretty plausible why Genie generated what it did imo, without having access to the inner workings of the underlying foundational model this is the best I can come up with.

My question was about if there were any strings that had wording similar to the "with a symmetrical relationship" message. You pointed to a variable name. So my question still stands.

You are right though with regards to data leakage into foundational models. This could be because of that.

Please see our new rules for submitting to SWE-bench: https://github.com/swe-bench/experiments?tab=readme-ov-file#reasoning-traces

If you want to be included in the leaderboard you should add reasoning trajectories to your submission.

Thanks

Understood @ofirpress – I'll commit our traj's in the morning. Once they're submitted does this mean we'll have to wait until next Monday to be eligible to be on the leaderboard, or can the trajectories be checked as part of this Monday's submission, given that at the time of us submitting the rules weren't as they are now?

To update we're still figuring out as a company how we feel about sharing trajectories particularly given the model is closed-source and is fine-tuned, so in the interests of keeping things tidy I'm going to close this PR until we make a decision on the subject.