MINDcraft is a groundbreaking open-source project that integrates large language models (LLMs) to control Minecraft bots, offering a new dimension of interaction and creativity within the game. It utilizes the Mineflayer library to enable these bots to perform complex tasks by generating high-level code. This project not only showcases the potential of AI in gaming but also pushes the boundaries of what can be achieved with language models in virtual environments. Novita AI is recommended as one of the project's API providers.

Project Overview of MINDcraft

At its core, MINDcraft is a node.js application that integrates LLMs to guide Minecraft bots in performing various tasks. The primary bot, named Andy, can communicate with players and autonomously set goals, such as gathering resources or constructing buildings. This capability allows Andy to play the game independently, continuously adapting and setting new objectives without human intervention. The project's architecture enables Andy to execute commands efficiently and interact with the game environment in a meaningful way.

Key Features

• Language Model Integration: MINDcraft employs LLMs to generate code that directs the bot's actions within Minecraft. This integration enables Andy to execute complex tasks efficiently by interpreting natural language instructions and translating them into actionable game commands.

• Autonomous Gameplay: Andy can independently set goals and navigate the game world, enhancing its ability to perform tasks without direct player input. This autonomy allows Andy to explore new strategies and adapt its behavior based on environmental feedback.

• Parameterized Commands: The project incorporates commands similar to those found in Toolformer, allowing Andy to execute tasks like resource collection more reliably. These parameterized commands simplify task execution by providing predefined actions that can be customized based on specific requirements.

MINDcraft Development and Implementation

Development

Inspiration and Initial Work

MINDcraft's development began after the release of a notable paper on Voyager, which introduced the Mineflayer library for creating Minecraft bots in Node.js. Unlike previous AI research that used image inputs and low-level outputs, Mineflayer allows AI systems to interact with Minecraft using high-level code, simplifying tasks like navigation and resource acquisition without needing reinforcement learning.

Leveraging LLMs for Instruction Following

Modern LLMs excel at code generation and instruction following, making it easy to condition an LLM to generate Mineflayer code based on human instructions, leading to the creation of Andy. Parameterized commands were implemented to enhance Andy's reliability in executing common skills. For example, Andy can output !collectBlocks("oak_log", 1) instead of generating complex code.

Enhancements and Future Directions

Over time, Andy's command set has been expanded, and automatic behaviors like avoiding aggressive mobs have been added. Recently, Andy can play Minecraft independently by setting natural language goals (e.g., "Collect materials to build a house"). Future plans include code reuse, experience reflection, improved spatial awareness, vision inputs, and multi-agent interactions.

Implementation

Agent Class and Main Logic Loop

The main class of MINDcraft is an Agent class that instantiates whenever Andy is not logged in. The main logic loop in handleMessage executes when a player sends a message to Andy:

handleMessage(message):
    history.add(message)
    while true:
        hist = history.getHistory()
        response = prompter.promptConvo(hist)
        if containsCommand(response):
            cleanChat(response)
            history.add(response)
            result = executeCommand(this, response)
            history.add(result)
        else:
            cleanChat(response)
            history.add(response)
            break

Handling Responses and Commands

Andy can return a single conversational response or iteratively call query or action commands. Queries provide world state information (e.g., !inventory, !nearbyBlocks ), while actions cause in-game behaviors (e.g., !followPlayer(name) , !collectBlocks(block) ). All actions are executed using the execute method in the Coder class.

Custom Code Generation

For complex behaviors like building, Andy can write custom code by outputting the newAction command to enter coding mode. Using the generateCode method from the Coder class, Andy writes custom JavaScript code with access to Mineflayer and a library of useful functions. The method includes a loop for multiple debugging attempts.

Role of In-Context Examples

In-context examples are crucial for the quality of Andy’s responses. An embedding model finds similar examples to the current history and adds them to the LLM context, ensuring Andy uses commands correctly and queries information before completing tasks.

How to Run MINDcraft

Requirements

• Minecraft Java Edition (up to v1.21.1, recommend v1.20.4)

• Node.js Installed (at least v14)

• One of these: OpenAI API Key | Gemini API Key | Anthropic API Key | Replicate API Key | Hugging Face API Key | Groq API Key | Ollama Installed. | Qwen API Key | Novita AI API Key |

Install and Run

Make sure you have the requirements above.

1. Clone or download this repository (big green button)

2. Rename keys.example.json to keys.json and fill in your API keys (you only need one). The desired model is set in andy.json or other profiles. For other models refer to the table below.

3. In terminal/command prompt, run npm install from the installed directory

4. Start a minecraft world and open it to LAN on localhost port 55916

5. Run node main.js from the installed directory

Customization

You can configure project details in settings.js. See file.You can configure the agent's name, model, and prompts in their profile like andy.json.

How to access the Novita AI API Key

Step 1: Go to novita.ai and log in

You can log in on Novita AI with Google or GitHub, which will create a new account on your first login. Alternatively, you can register using your email address.

Step 2: Manage Novita AI LLM API Key

Novita AI secures API access through Bearer authentication by including an API Key in the request header, such as “Authorization: Bearer {API Key}”.To manage your keys, navigate to “Key Management” in the settings. A default key is automatically generated upon your first login. You can create additional keys by clicking “+ Add New Key”.

Step 3: Top Up Credit

Novita AI offers new users a voucher with some credit to try our products. To add more credit, please visit Billing and Payments and follow the guide on Payment Methods.

Useful Links and Resources

• Github Repository

• FAQ

• Discord

• Kolby’s Twitter

• Max’s Twitter

Conclusion

MINDcraft represents a significant step forward in AI research within gaming environments. By harnessing the power of LLMs, it opens new possibilities for creativity and interaction in Minecraft. As the project continues to evolve, community participation and feedback will play a crucial role in shaping its future developments. The project's open-source nature encourages collaboration among developers and researchers interested in exploring the intersection of AI and gaming.

Originally published at Novita AI

Novita AI is the All-in-one cloud platform that empowers your AI ambitions. Integrated APIs, serverless, GPU Instance — the cost-effective tools you need. Eliminate infrastructure, start free, and make your AI vision a reality.