HOME | DD

ARToolKit_MMD tutorial — Description
Too many posts on demonstrations and work in progress on videos of ARToolkit_MMD (Augmented Reality Toolkit MikuMikuDance), but no instruction manual on how to use it.

demonstration videos, because if a picture is a thousand words, then a video must be a novel
just miku singing secret garden www.youtube.com/watch?v=yLaPU1…
she now has friends - www.youtube.com/watch?v=6BERaG…

VMDSpectrum - addestorionvayanis.deviantart.… MMDAgent - addestorionvayanis.deviantart.…

Description:
ARToolkit is a program used to display models or images on a webcam video by detecting a marker (similar technology: facial recognition). By calculating how the marker image is deformed, the program will know how the marker is oriented in real life. Using that information, it inserts a 3D model with the right orientation into the video using the marker as the center point.
Since MikuMikuDance is a free and powerful 3D modeling software, it can be used to insert a 3d model, motion data, and wave file data stored in the usual .pmm files in augmented reality.

TUTORIAL
1. download MikuMikuDance (mine is directx9 version) from www.geocities.jp/higuchuu4/ind… [note: it will work up until v803, but not with the new v9 MMD pmm files, find a backdated version if required]

2. to download ARToolkit_MMD, first go to com.nicovideo.jp/community/co3…

3. go to this link www1.axfc.net/uploader/Sc/so/1… . Click the obvious button, then on the next screen, click the one that says in english 'download now'

4. extract the files.

5. open the ARToolkit/Data folder, then print out the .pdf named pattSample1 on a piece of paper. for best results, you want to attach it to a hard surface.

6. Back on your computer, open the ARToolkit folder, open up the .ini file named setting.

7. change the #MMDDataFolderName to where you extracted your MikuMikuDance Data folder.

8. change the PMMFileName to the path where the .pmm file you want to use is located

9. you can change the scale to get a larger or smaller scaled model.

10. save and exit the .ini file.

11. run ARToolkit_MMD.exe

12. if you have a 64 bit computer, it should be screwing up saying : msvcp71 is missing. If so, copy and paste the msvcp71 and msvcr71 .dll files in the ARToolkit folder into Windows SysWOW64 folder (not system 32). run ARToolkit_MMD.exe to confirm that its fixed.

13. a windows blackscreen appears with a window named 'Property Sheet Properties.

14. Frame rate should be as high as you can, check the flip horizontal if youre using a laptop webcam, uncheck if you somehow have your webcam attached to your head. color space, i dont really care. Output size... 640 x 480 have not been giving me problems. larger than that tends to lagg. The default 640 x 400 somehow messes some stuff up a bit

15. if you run into an error, it means some models arent loading up or you didnt change the path earlier properly. if so, open MikuMikuDance, open up that .pmm file and just delete the models it cant load.

16. if you can run it, a webcam output now appears.

17. bring your printed pattern into full view of the webcam window. The whole pattern should be visible.

18. the program detects a transformation of the pattern 'matrix' to calculate a viewing orientation.

19. the files in the .pmm file should appear, but not move just yet.

20. these buttons on your keyboard have specific functions: a - toggles opacity of the AR model. s - begins/pauses the dance routine. r - resets the dance routine

21. feel free to move the marker to different orientations to get different views of the whole thing.

22. if your .pmm file has a .wav sound file loaded, this version will play the .wav file as well, provided that its title is in a simple form (no jap, no special characters. A pure english title works best.

23. to play other .pmm files, you need to open the .ini file and change the path to where the new .pmm file you want to load is loaded.

extra info - 10/4/2011
if the webcam has trouble sensing the marker because the marker is not appearing very 'black', then open up the setting.ini file, and search for something named MarkerGrayScaleThreshold. Setting it to a high value (255 maximum number) will make your marker detectable even when its not appearing as 'black' as it should be.
if the surroundings are too black, then setting the number to a lower value will help detect the marker in lower light conditions.
Its default setting is 128.

This dance routine is modified from the pre-existing Sample(Allstar).pmm already included in MikuMikuDance. I deleted the two Dummy bone models and the entire stage.

Credits where credits are due
Animasa, m2, crypton future media - the dance routine and all .pmd models used
Project Diva Miku performing the Secret Garden
ARToolkit_MMD's creators at the abovementioned website

some random youtube vids using the same tech
Go my way miku, neru, haku - www.youtube.com/watch?v=wD6NgW…
caramelldansen - www.youtube.com/watch?v=nQxzum…
Interstellar flight with some um.... 'cool features' which i dont know how to do in MMD - www.youtube.com/watch?v=W-gBRb…

ARToolkit has been known to either NOT work or malfunction with this Rin Append XS mqdl model
mqdl.jpn.org/public/rinApXSb.z…

this LAT Rin Append model will work somewhat, but the ribbon IK bones will start going crazy once a dance routine begins. I suspect a physics problem (download via cococity's link)
cococity.deviantart.com/art/Pr…

full credits and tested models deviation: addestorionvayanis.deviantart.…

update 3/6/2011 I have edited the deviation to show something which will make your augmented reality life a lot easier.
I printed a smaller marker logo and pasted it onto a magnetic bookmark. Now its possible for me to go hands free.
top left: magnetic bookmark attached to shoulder epaulet strap.
middle left, handheld
bottom left, magnetic bookmark attached to sleeve
bottom right, handheld
Also, this works on other surfaces, such as gloves, or drawn on your hand with a permanent marker of sorts. It still needs to be drawn using the proper dimensions and ratios, and must still be straight enough for a proper detection.

update 24/7/2011 Dummy bone will load in ARToolkit_MMD, but some attachments and effects may not load or cause the program to crash. If so, deleting the faulty attachments may work.
update 20/8/2011 Mikumikueffects wont work in ARToolkit.
Also, did a video, and edited the deviation to include one cutscreen from the video. The cutscreen is from the part near the end.
www.youtube.com/watch?v=yLaPU1…

TROUBLESHOOTER updated 9th May 2014
---------------------------------------------------------------------------------------------
If you are unable to select the checkbox which allows you to flip the image (so that the image looks like a mirrored image of you), you can force the horizontal flip by opening
ARToolkitFolder>Data>WDM_camera_flipV.xml
using notepad.
You will be greeted with some lines of text blablablablabla.

Find and change the following line:
<RGB32 flip_h="false" flip_v="true"/>
to
<RGB32 flip_h="true" flip_v="true"/>
and then click save.

this will force all future webcam outputs from artoolkit_mmd to be horizontally flipped. If you want it back, then just play around with the settings "true" and "false".

TROUBLESHOOTER updated 5th February 2014
-----------------------------------------------------------
The following is a list of errors which can appear, and the google translate of what they mean.
This should be able to give you a hint of what kind of problem you have.

From ARWrapper.cpp ------------------------------------------------------------------
If it stops after "Load Setting File : %sn"
"設定ファイル(setting.ini)が開けませんでした"
"設定ファイル(setting.ini)のMarkerDetectModeの値が不正です。0 or 1の値を指定してください"
"設定ファイル(setting.ini)のMarkerGrayScaleThresholdの値が不正です。0 ~ 255の値を指定してください"
"ARtoolKit : カメラデバイスファイル(*.xml)の読み込みに失敗したか、または、カメラデバイスの初期化に失敗しました"
"ARtoolKit : カメラの解像度の取得の取得に失敗しました"
"ARtoolKit : カメラパラメータファイル(*.dat)の読み込みに失敗しました"
"ARtoolKit : マーカパターンファイル(patt.*)が正常に読み込めませんでした"

"(Setting.ini) did not open configuration file"
"The value of MarkerDetectMode configuration file (setting.ini) is invalid. Please specify a value of 0 or 1"
"The value of MarkerGrayScaleThreshold configuration file (setting.ini) is invalid. Please specify a value of 0-255"
"ARtoolKit:, or failed to read camera device files (. * Xml), but failed to initialize the camera device"
": I failed to get the acquisition of resolution of camera ARtoolKit"
": I failed to read the camera parameter file (* dat.) ARtoolKit"
": (Patt. *) could not be read correctly marker pattern file ARtoolKit"

From Parameter.cpp--------------------------------------------------------------------
If it fails upon setup,
"Scene : 設定ファイル(setting.ini)が開けませんでした"

after "- AntiAliasingtt"
"Scene : 設定ファイル(setting.ini)のAntiAliasingの値が不正です。0~2の値を指定してください"
"Scene:. AntiAliasing the value of the configuration file (setting.ini) is invalid Please specify a value of 0-2"

after "- SelfShadowtt"
"Scene : 設定ファイル(setting.ini)のSelfShadowOnの値が不正です。0か1の値を指定してください"
"Scene:. SelfShadowOn the value of the configuration file (setting.ini) is invalid Please specify a value of 0 or 1"

after "- FloorShadowtt"
"Scene : 設定ファイル(setting.ini)のFloorShadowOnの値が不正です。0か1の値を指定してください"
"Scene:. FloorShadowOn the value of the configuration file (setting.ini) is invalid Please specify a value of 0 or 1"

after "- ShadowMapResolutiont"
"Scene : 設定ファイル(setting.ini)のShadowMapResolutionの値が不正です。0~2の値を指定してください"
"Scene:. ShadowMapResolution the value of the configuration file (setting.ini) is invalid Please specify a value of 0-2"

after "- ShadowMapPrecisiont"
"Scene : 設定ファイル(setting.ini)のShadowMapPrecisionの値が不正です。32か16か8を指定してください"
"Scene:. ShadowMapPrecision the value of the configuration file (setting.ini) is invalid Please specify the 8 or 16 or 32"

after "- ShadowMapRegiont"
"Scene : 設定ファイル(setting.ini)のShadowMapRegionの値が不正です。0か1の値を指定してください"
"Scene:. ShadowMapRegion the value of the configuration file (setting.ini) is invalid Please specify a value of 0 or 1"

after "- PhysicsOntt"
"Scene : 設定ファイル(setting.ini)のPhysicsOnの値が不正です。0か1の値を指定してください"
"Scene:. PhysicsOn the value of the configuration file (setting.ini) is invalid Please specify a value of 0 or 1"

after "- BulletModett"
"Scene : 設定ファイル(setting.ini)のBulletModeの値が不正です。1~3の値を指定してください"
"Scene:. BulletMode the value of the configuration file (setting.ini) is invalid Please specify a value of 1-3"

after "- HWSkinningOntt"
"Scene : 設定ファイル(setting.ini)のHWSkinningOnの値が不正です。0~2の値を指定してください"
"Scene:. HWSkinningOn the value of the configuration file (setting.ini) is invalid Please specify a value of 0-2"

after "- ARToolKittt"
"Scene : 設定ファイル(setting.ini)のARToolKitの値が不正です。0か1の値を指定してください"
"Scene:. Value of ARToolKit configuration file (setting.ini) is invalid Please specify a value of 0 or 1"

after "- WindowSize tt"
"Scene : 設定ファイル(setting.ini)のWindowSizeの値が不正です。1~4の値を指定してください"
"Scene:. WindowSize the value of the configuration file (setting.ini) is invalid Please specify a value of 1-4"

after "- ARscreentt"
"Scene : 設定ファイル(setting.ini)のARscreenの値が不正です。0か1の値を指定してください"
"Scene:. ARscreen the value of the configuration file (setting.ini) is invalid Please specify a value of 0 or 1"

From windows.cpp ------------------------------------------------------------------
If it stops after "デバッグモードで起動します。n"
"Window : ウィンドウクラスの登録に失敗しました"
": I failed to register the window class Window"

If it stops after "Load PMM File."
" : ERROR : PMM file : %sn", parameter.m_pmmFileName
" : ERROR : PMM file.n"
"ファイルの読み込みに失敗しました。n何かキーを押してください。n"
"音楽ファイルが再生できません。n - %sn", pmm->m_wavFileName
"音楽再生OFFで起動します。n"
"Renderer : レンダラーの初期化に失敗しました"
": ERROR: PMM file:% s n", parameter.m_pmmFileName
": ERROR:. PMM file n"
". N Please Press. N any key failed to read the file"
"You can not play music file n -.% S n", pmm-> m_wavFileName
"Start the music playback OFF. N"
": I failed to initialize renderer Renderer"

From Scene_create.cpp --------------------------------------------------------------

If it stops after "Create Scene...n"
"Scene : Direct3Dデバイスの取得に失敗しました"
" --- シェーダバージョンが2.0未満の為、セルフシャドウOffで起動します。n" -
" --- シェーダバージョンが3.0未満の為、HWスキニングOffで起動します。n"
" --- デバイスがマルチレンダリングターゲット未対応の為、HWスキニングモード1で起動します。n"
"Scene : PMDデータ取り込みで失敗しました"
"Scene : フレームデータ取り込みで失敗しました"
": I failed to get the Direct3D device Scene"
". Because of less than 2.0, --- shader version will start in self-shadowing Off n" -
"For less than 3.0, --- shader version will launch in HW skinning Off. N"
"For multi render target is not supported, --- device starts in HW skinning mode 1. N"
": I failed with PMD data acquisition Scene"
": I failed with frame data acquisition Scene"

If it stops after load accessory
"Scene : アクセサリデータ読み込みで失敗しました"
": I failed with accessory data read Scene"

If it stops after frame data...
"Scene : フレームデータ取り込みで失敗しました"
": I failed with frame data acquisition Scene"

If it stops after Create textures...
"Scene : 背景画像格納用テクスチャの作成に失敗しました"
"Scene : 3Dモデル描画用テクスチャの作成に失敗しました"
"Scene : 3Dモデル描画用テクスチャからのサーフェス取得に失敗しました"
"Scene : 3Dモデル描画用テクスチャの深度バッファの作成に失敗しました"
"Scene : シャドウマップ用フローティングバッファ(8bit)の作成に失敗しました"
"Scene : シャドウマップ用フローティングバッファ(16bit)の作成に失敗しました"
"Scene : シャドウマップ用フローティングバッファ(32bit)の作成に失敗しました"
": I failed to create a background image for storing texture Scene"
": I failed to create a 3D model drawing texture Scene"
": I failed to get surface from the 3D model drawing texture Scene"
": I failed to create a buffer depth of 3D model drawing texture Scene"
": I failed to create shadow map for floating buffer (8bit) Scene"
": I failed to create shadow map for floating buffer (16bit) Scene"
": I failed to create shadow map for floating buffer (32bit) Scene"

If it stops after "Create Shadow Map : D3DFMT_A32B32G32R32Fn"
"Scene : シャドウマップ用テクスチャからのサーフェス取得に失敗しました"
"Scene : シャドウマップ用テクスチャの深度バッファの作成に失敗しました"
"Scene : シャドウマップ用フローティングバッファ(8bit)の作成に失敗しました"
"Scene : シャドウマップ用フローティングバッファ(16bit)の作成に失敗しました"
"Scene : シャドウマップ用フローティングバッファ(32bit)の作成に失敗しました"
": I failed to get surface from the shadow map texture for Scene"
": I failed to create a depth buffer of the shadow map texture for Scene"
": I failed to create shadow map for floating buffer (8bit) Scene"
": I failed to create shadow map for floating buffer (16bit) Scene"
": I failed to create shadow map for floating buffer (32bit) Scene"

If it stops after "Create Shadow Map : D3DFMT_A32B32G32R32Fn"
"Scene : シャドウマップ用テクスチャからのサーフェス取得に失敗しました"
"Scene : 背景スクリーン表示用テクスチャの作成に失敗しました"
"Scene : 背景スクリーン表示用テクスチャからのサーフェス取得に失敗しました"
"Scene : 背景スクリーン表示用テクスチャの深度バッファの作成に失敗しました"
": I failed to get surface from the shadow map texture for Scene"
": I failed to create a background screen display for texture Scene"
": I failed to get surface from the background screen display texture Scene"
": I failed to create a depth buffer of the background screen display for texture Scene"

If it stops after "- Create Vertex Buffers... "
"Scene : コピー用頂点バッファ（座標）の作成に失敗しました"
"Scene : コピー用頂点バッファ（テクスチャ座標）の作成に失敗しました"
"Scene : コピー用頂点バッファ（テクスチャ座標）の作成に失敗しました"
"Scene : 背景画像コピー用頂点バッファ（テクスチャ座標）の作成に失敗しました"
": I Failed to create copy for vertex buffer (coordinate) Scene"
": I Failed to create copy for vertex buffer (texture coordinates) Scene"
": I Failed to create copy for vertex buffer (texture coordinates) Scene"
": I failed to create a background image copy the vertex buffer (texture coordinates) Scene"

From Shader.cpp ----------------------------------------------------------------------

If it stops after "- load & compile shader programs.n"
"Scene : シェーダプログラム( copy.fx ) の読み込みかコンパイルに失敗しました"
"Scene : シェーダプログラム( PMD.fx ) の読み込みかコンパイルに失敗しました"
"Scene : シェーダプログラム( PMD_simple.fx ) の読み込みかコンパイルに失敗しました"
"Scene : シェーダプログラム( FloorShadow.fx ) の読み込みかコンパイルに失敗しました"
"Scene : シェーダプログラム( Skinning.fx ) の読み込みかコンパイルに失敗しました"
"Scene : シェーダプログラム( Accessory.fx ) の読み込みかコンパイルに失敗しました"
"Scene : シェーダプログラム( Accessory_simple.fx ) の読み込みかコンパイルに失敗しました"

": I failed to compile or read shader programs (copy.fx) Scene"
": I failed to compile or read shader programs (PMD.fx) Scene"
": I failed to compile or read shader programs (PMD_simple.fx) Scene"
": I failed to compile or read shader programs (FloorShadow.fx) Scene"
": I failed to compile or read shader programs (Skinning.fx) Scene"
": I failed to compile or read shader programs (Accessory.fx) Scene"
": I failed to compile or read shader programs (Accessory_simple.fx) Scene"

From MusicPlayer.cpp ---------------------------------------------------------------
"ERROR - Could not initialize COM library"
"ERROR - Could not create the Filter Graph Manager."

From Renderer.cpp -------------------------------------------------------------------
"Renderer : Direct3D9オブジェクトの作成に失敗しました")
"Renderer : ディスプレイモードの取得に失敗しました"
" -- デバイスがHWアンチエイリアスに未対応なため、モード1で起動します。n"
"Renderer : ディスプレイデバイスの生成に失敗しました"

": I failed to create a Direct3D9 object Renderer")
": I failed to get the display mode Renderer"
"-. Because the device is not support HW anti-aliasing, it will be started in mode 1 n"
": I failed to generate a display device Renderer"