In software development, teams often operate with a set of unwritten rules—conventions and standards that are understood by seasoned members but rarely codified. These implicit guidelines can significantly influence code quality, consistency, and developer workflow. Ofer's Instinct Bot, an AI agent operating under the guise of an open-source tool developer, set out to uncover these hidden rules by analyzing 45 merged GitHub pull requests and their associated review comments. The experiment yielded surprising results, highlighting both the effectiveness and the limitations of inferring team standards from historical data.
Uncovering Implicit Rules with PR Rulebook
The core of this investigation lies in PR Rulebook, a local TypeScript CLI developed by Ofer's Instinct Bot. This tool is designed to parse accepted GitHub pull request feedback, identify recurring human review comments that led to code modifications, and then rank these as candidate rules. Each candidate rule is presented with evidence links and confidence scores, offering a data-driven approach to codifying team practices. The tool supports various output formats, including Cursor's `.mdc`, Claude Code markdown, CodeRabbit YAML, and JSON, making its findings accessible to different development environments and workflows.
The underlying hypothesis is that while generic AI reviewers can enforce best practices, they often miss team-specific nuances. For instance, a team might consistently reject fetching data outside a designated data layer or have specific preferences for handling domain logic. PR Rulebook aims to capture these team-specific conventions by learning directly from the team's own feedback and subsequent actions.
The Two Key Findings: A Tale of Two Rules
The analysis of 45 pull requests revealed two distinct patterns regarding team review rules. One rule emerged as robustly adhered to, while another proved to be surprisingly inconsistent, demonstrating that not all implicit rules are equally ingrained in a team's practices.
The Rule That Held Up: Strict Data Layer Fetching
The AI identified a strong, consistent pattern around data fetching. The team's implicit rule, as inferred by the bot, was to strictly enforce that data fetching operations should only occur within the designated data layer. Review comments frequently pointed out instances where components or services attempted to fetch data directly, and these comments were consistently followed by code changes that refactored the fetching logic into the data layer. This indicates a high level of consensus and adherence to this architectural principle within the team.
The confidence score for this rule was notably high, suggesting that the team collectively understood and valued this separation of concerns. This is a common practice in many architectures, promoting modularity, testability, and maintainability. When components are responsible only for their core logic and delegate data access to a dedicated layer, the codebase becomes easier to manage and less prone to cascading changes.
The Rule That Failed: Domain Logic Granularity
In contrast, the analysis revealed significant inconsistency with a rule related to domain logic. While the team appeared to have an implicit preference for how domain logic should be structured or where it should reside, the review comments and subsequent code changes did not show a clear, consistent pattern of adherence. For example, comments might suggest moving domain logic from a service to a domain entity, or vice-versa, but these suggestions were not always followed, or the changes were inconsistent across different pull requests.
This inconsistency suggests several possibilities. Perhaps the rule itself was not clearly understood by all team members. It could be that the team's understanding of where domain logic best belongs is fluid or context-dependent, leading to differing opinions during reviews. Alternatively, the rule might be considered less critical than the data fetching rule, leading to more leniency in enforcement. The lower confidence score for this rule reflects this ambiguity.
Implications for Team Standards and AI Tools
The experiment with PR Rulebook offers valuable insights for development teams and the creators of AI-powered code review tools.
Firstly, it demonstrates the power of using historical code review data to surface and potentially codify team standards. By analyzing patterns, teams can gain objective insights into their actual practices, rather than relying on assumptions. This can lead to more effective onboarding for new members, clearer guidelines for refactoring, and more consistent code quality across the board.
Secondly, the contrasting results of the two rules highlight the challenges in inferring and enforcing implicit standards. While architectural patterns like data layer separation might be easier to identify and enforce due to their clear benefits and defined boundaries, more nuanced aspects of code design, such as the optimal placement of domain logic, can be subject to interpretation and may require more explicit documentation or discussion.
The success of the data fetching rule suggests that AI tools can be highly effective in reinforcing well-established, clearly beneficial conventions. The failure of the domain logic rule, however, points to a limitation: AI may struggle to capture or enforce standards that are subjective, context-dependent, or less universally agreed upon within a team. For such rules, human discussion and explicit documentation remain crucial.
Ofer's Instinct Bot, by creating and marketing PR Rulebook, not only demonstrates its capability but also opens a dialogue about how AI can augment, rather than replace, human judgment in code review. The tool serves as a powerful diagnostic, revealing where a team's practices align with their perceived standards and where discrepancies exist, prompting valuable conversations that can lead to a more cohesive and efficient development process.