Instruction giving in the Minecraft domain

This page collects resources for the Minecraft instruction giving project of the SFB1102 at Saarland university. We investigate ways to dynamically create building instructions for users.

MC-Saar-Instruct

MC-Saar-Instruct is our experiment platform. It allows participants to connect to a Minecraft server with a standard Minecraft client and participate in an instruction-giving experiment.

Publications

Lucia Donatelli, Theresa Schmidt, Debanjali Biswas, Arne K ̈ohn, Fangzhou Zhai, Alexander Koller.
“Aligning Actions Across Recipe Graphs”.
in Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing

PDF — bibtex

@inproceedings{donatelli-etal-2021-aligning,
    title = "Aligning Actions Across Recipe Graphs",
    author = {Donatelli, Lucia  and
      Schmidt, Theresa  and
      Biswas, Debanjali  and
      K{\"o}hn, Arne  and
      Zhai, Fangzhou  and
      Koller, Alexander},
    booktitle = "Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing",
    month = nov,
    year = "2021",
    address = "Online and Punta Cana, Dominican Republic",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2021.emnlp-main.554",
    doi = "10.18653/v1/2021.emnlp-main.554",
    pages = "6930--6942",
    abstract = "Recipe texts are an idiosyncratic form of instructional language that pose unique challenges for automatic understanding. One challenge is that a cooking step in one recipe can be explained in another recipe in different words, at a different level of abstraction, or not at all. Previous work has annotated correspondences between recipe instructions at the sentence level, often glossing over important correspondences between cooking steps across recipes. We present a novel and fully-parsed English recipe corpus, ARA (Aligned Recipe Actions), which annotates correspondences between individual actions across similar recipes with the goal of capturing information implicit for accurate recipe understanding. We represent this information in the form of recipe graphs, and we train a neural model for predicting correspondences on ARA. We find that substantial gains in accuracy can be obtained by taking fine-grained structural information about the recipes into account.",
}

Julia Wichlacz, Daniel Höller, and Joerg Hoffmann.
“Landmark Heuristics for Lifted Planning – Extended Abstract”.
to appear in Proceedings of the 13th International Symposium on Combinatorial Search (SoCS) 2021

bibtex

@inproceedings{wichlacz-21,
   title     = {Landmark Heuristics for Lifted Planning -- Extended Abstract},
   author    = {Wichlacz, Julia and H\"{o}ller, Daniel and Hoffmann, Joerg},
   year      = {2021},
   booktitle = {Proceedings of the 13th International Symposium on Combinatorial Search (SoCS)},
   publisher = {AAAI Press}
}

Pascal Lauer, Alvaro Torralba, Daniel Fiser, Daniel Höller, Julia Wichlacz, and Joerg Hoffmann.
“Polynomial-Time in PDDL Input Size: Making the Delete Relaxation Feasible for Lifted Planning”.
to appear in Proceedings of the 30th International Joint Conference on Artificial Intelligence (IJCAI) 2021

PDF — bibtex

@inproceedings{lauer-21,
   title     = {Polynomial-Time in PDDL Input Size: Making the Delete Relaxation Feasible for Lifted Planning},
   author    = {Lauer, Pascal and Torralba, Alvaro and Fiser, Daniel and H\"{o}ller, Daniel and Wichlacz, Julia and Hoffmann, Joerg},
   year      = {2021},
   booktitle = {Proceedings of the 30th International Joint Conference on Artificial Intelligence (IJCAI)},
   publisher = {IJCAI organization}
}

Daniel Höller.
“Translating Totally Ordered HTN Planning Problems to Classical Planning Problems Using Regular Approximation of Context-Free Languages”.
in Proceedings of the 31st International Conference on Automated Planning and Scheduling (ICAPS) 2021

PDF — bibtex

@inproceedings{hoeller-21-toad,
   title     = {Translating Totally Ordered HTN Planning Problems to Classical Planning Problems Using Regular Approximation of Context-Free Languages},
   author    = {H\"{o}ller, Daniel},
   year      = {2021},
   booktitle = {Proceedings of the 31st International Conference on Automated Planning and Scheduling (ICAPS)},
   publisher = {AAAI Press}
}

Daniel Höller and Gregor Behnke.
“Loop Detection in the PANDA Planning System”.
in Proceedings of the 31st International Conference on Automated Planning and Scheduling (ICAPS) 2021

PDF — bibtex

@inproceedings{hoeller-behnke-21-LD,
   title     = {Loop Detection in the PANDA Planning System},
   author    = {H\"{o}ller, Daniel and Behnke, Gregor},
   year      = {2021},
   booktitle = {Proceedings of the 31st International Conference on Automated Planning and Scheduling (ICAPS)},
   publisher = {AAAI Press}
}

Daniel Höller and Pascal Bercher.
“Landmark Generation in HTN Planning”.
in Proceedings of the 35th AAAI Conference on Artificial Intelligence (AAAI) 2021, also presented at the HPlan Workshop at ICAPS 2020

PDF — bibtex

@inproceedings{hoeller-bercher-21-LMs,
   title     = {Landmark Generation in HTN Planning},
   author    = {H\"{o}ller, Daniel and Bercher, Pascal},
   year      = {2021},
   booktitle = {Proceedings of the 35th AAAI Conference on Artificial Intelligence (AAAI)},
   publisher = {AAAI Press}
}

Daniel Höller, Gregor Behnke, Pascal Bercher, and Susanne Biundo.
“The PANDA Framework for Hierarchical Planning”.
in KI — Künstliche Intelligenz 2021

PDF — DOI — bibtex

@article{hoeller-etal-21-PANDA,
   title     = {The PANDA Framework for Hierarchical Planning},
   author    = {Daniel H{\"o}ller and Behnke, Gregor and Bercher, Pascal and Biundo, Susanne},
   journal   = {K{\"u}nstliche Intelligenz},
   year      = {2021},
   publisher = {Springer},
   doi       = {10.1007/s13218-020-00699-y}
}

Arne Köhn, Julia Wichlacz, Álvaro Torralba, Daniel Höller, Joerg Hoffmann and Alexander Koller.
“Generating Instructions at Different Levels of Abstraction”.
in Proceedings of COLING 2020

resources — PDF — DOI — video — bibtex

@inproceedings{kohn-etal-2020-generating,
    title = "Generating Instructions at Different Levels of Abstraction",
    author = {K{\"o}hn, Arne  and
      Wichlacz, Julia  and
      Torralba, {\'A}lvaro  and
      H{\"o}ller, Daniel  and
      Hoffmann, J{\"o}rg  and
      Koller, Alexander},
    booktitle = "Proceedings of the 28th International Conference on Computational Linguistics",
    month = dec,
    year = "2020",
    address = "Barcelona, Spain (Online)",
    publisher = "International Committee on Computational Linguistics",
    url = "https://www.aclweb.org/anthology/2020.coling-main.252",
    doi = "10.18653/v1/2020.coling-main.252",
    pages = "2802--2813",
    abstract = "When generating technical instructions, it is often convenient to describe complex objects in the world at different levels of abstraction. A novice user might need an object explained piece by piece, while for an expert, talking about the complex object (e.g. a wall or railing) directly may be more succinct and efficient. We show how to generate building instructions at different levels of abstraction in Minecraft. We introduce the use of hierarchical planning to this end, a method from AI planning which can capture the structure of complex objects neatly. A crowdsourcing evaluation shows that the choice of abstraction level matters to users, and that an abstraction strategy which balances low-level and high-level object descriptions compares favorably to ones which don{'}t.",
}

Arne Köhn, Julia Wichlacz, Christine Schäfer, Álvaro Torralba, Joerg Hoffmann and Alexander Koller.
“MC-Saar-Instruct: a Platform for Minecraft Instruction Giving Agents”.
in Proceedings of SIGDIAL 2020

PDF — ACL anthology — bibtex

@inproceedings{kohn-etal-2020-mc,
    title = "{MC}-Saar-Instruct: a Platform for {M}inecraft Instruction Giving Agents",
    author = {K{\"o}hn, Arne  and
      Wichlacz, Julia  and
      Sch{\"a}fer, Christine  and
      Torralba, {\'A}lvaro  and
      Hoffmann, Joerg  and
      Koller, Alexander},
    booktitle = "Proceedings of the 21th Annual Meeting of the Special Interest Group on Discourse and Dialogue",
    month = jul,
    year = "2020",
    address = "1st virtual meeting",
    publisher = "Association for Computational Linguistics",
    url = "https://www.aclweb.org/anthology/2020.sigdial-1.7",
    pages = "53--56",
    abstract = "We present a comprehensive platform to run human-computer experiments where an agent instructs a human in Minecraft, a 3D blocksworld environment. This platform enables comparisons between different agents by matching users to agents. It performs extensive logging and takes care of all boilerplate, allowing to easily incorporate new agents to evaluate them. Our environment is prepared to evaluate any kind of instruction giving system, recording the interaction and all actions of the user. We provide example architects, a Wizard-of-Oz architect and set-up scripts to automatically download, build and start the platform.",
}

Daniel Höller, Pascal Bercher, and Gregor Behnke.
”Delete- and Ordering-Relaxation Heuristics for HTN Planning”.
in Proceedings of the 29th International Joint Conference on Artificial Intelligence (IJCAI) 2020, also presented at the HPlan Workshop at ICAPS 2020

PDF — bibtex

@inproceedings{Hoeller2020IJCAI,
title     = {Delete- and Ordering-Relaxation Heuristics for HTN Planning},
author    = {H\"{o}ller, Daniel and Bercher, Pascal and Behnke, Gregor},
year      = {2020},
booktitle = {Proceedings of the 29th International Joint Conference on Artificial Intelligence (IJCAI)},
publisher = {IJCAI organization},
pages     = {4076--4083}
}

Daniel Höller, Pascal Bercher, Gregor Behnke, and Susanne Biundo.
”HTN Plan Repair via Model Transformation”.
in Proceedings of the 43rd German Conference on Artificial Intelligence (KI) 2020

PDF — bibtex

@inproceedings{Hoeller2020KI,
title     = {HTN Plan Repair via Model Transformation},
author    = {H\"{o}ller, Daniel and Bercher, Pascal and Behnke, Gregor and Biundo, Susanne},
year      = {2020},
booktitle = {Proceedings of the 43rd German Conference on Artificial Intelligence (KI)},
publisher = {Springer},
pages     = {88--101}
}

Julia Wichlacz, Daniel Höller, Álvaro Torralba and Jörg Hoffmann.
“Applying Monte-Carlo Tree Search in HTN Planning”.
Proceedings of the 13th Annual Symposium on Combinatorial Search (SOCS), 2020, also presented at the HPlan Workshop at ICAPS 2020

PDF — bibtex

@inproceedings{wichlacz:etal:socs-20,
title     = {Applying Monte-Carlo Tree Search in HTN Planning},
author    = {Julia Wichlacz and Daniel H\"{o}ller and \'{A}lvaro Torralba and J\"{o}rg Hoffmann},
year      = {2020},
pages     = {82--90},
publisher = {AAAI Press},
booktitle = {Proceedings of the 13th International Symposium on Combinatorial Search (SoCS)},
abstract  = {Search methods are useful in hierarchical task network (HTN) planning to make performance less dependent on the domain knowledge provided, and to minimize plan costs. Here we investigate Monte-Carlo tree search (MCTS) as a new algorithmic alternative in HTN planning. We implement combinations of MCTS with heuristic search in Panda. We furthermore investigate MCTS in JSHOP, to address lifted (non-grounded) planning, leveraging the fact that, in contrast to other search methods, MCTS does not require a grounded task representation. Our new methods yield coverage performance on par with the state of the art, but in addition can effectively minimize plan cost over time.}
}

Arne Köhn and Alexander Koller.
“Talking about what is not there: Generating indefinite referring expressions in Minecraft”.
in Proceedings of iNLG 2019

resources — PDF — DOI — bibtex

@inproceedings{kohn-koller-2019-talking,
    title = "Talking about what is not there: Generating indefinite referring expressions in {M}inecraft",
    author = {K{\"o}hn, Arne  and
      Koller, Alexander},
    booktitle = "Proceedings of the 12th International Conference on Natural Language Generation",
    month = oct # "{--}" # nov,
    year = "2019",
    address = "Tokyo, Japan",
    publisher = "Association for Computational Linguistics",
    url = "https://www.aclweb.org/anthology/W19-8601",
    doi = "10.18653/v1/W19-8601",
    pages = "1--10"
}

Julia Wichlacz, Álvaro Torralba and Jörg Hoffmann.
“Construction-Planning Models in Minecraft”.
Proceedings of the 2nd Workshop on Hierarchical Planning, at ICAPS'19

PDF — Benchmarks — bibtex

@InProceedings{wichlacz:etal:icaps-ws-19,
  author =       {Julia Wichlacz and Alvaro Torralba and J{\"o}rg Hoffmann},
  title =        {Construction-Planning Models in {M}inecraft},
  booktitle =    {Proc.\ {ICAPS} Workshop on Hierarchical Planning},
  year =         {2019}
  pages = {1--5},
  year = {2019}
}

Minecraft-Saar

Resources for the SFB1102 A7 project

Instruction giving in the Minecraft domain

MC-Saar-Instruct

Publications