Japanese immigrants first came to Bainbridge Island in the 1880s, working in sawmills and strawberry harvesting, and by the 1940s had become an integral part of the island's community. Because of the island's proximity to naval bases, local Japanese Americans were the first in the country to be interned; 227 Japanese Americans were ordered to leave the island with six days' notice. They departed by ferry on March 30, 1942. Most internees were sent to Manzanar, CA, though some were later transferred to Minidoka, Idaho. Local newspapers such as the The Bainbridge Review (made famous by the novel and film Snow Falling on Cedars) spoke out against the internment and continued to publish correspondence from internees. A Seattle Post-Intelligencer photograph of Bainbridge Island resident Fumiko Hayashida and her 13-month-old daughter preparing to board the ferry that day became famous as a symbol of the internment. About 150 returned to the island after the end of World War II. By 2011, about 90 survivors remained, of whom 20 still lived on the island.
The first part of the memorial to be constructed was an outdoor cedar "story wall" with the names of all 276 Japanese and Japanese Americans resident on the island at the time. The wall was designed by local architect Johnpaul Jones, an American Indian and the principal of Jones and Jones Architects. The grounds of the memorial wall is natural landscaping, native species including sword fern, mahonia, salal, and shore pine. Local artist Steve Gardner created friezes to be placed on the wall, depicting some of the scenes of residents being herded onto the ferries; he stated that the project "sucked me in in a way I hadn't thought about. This really is a story about American citizens." The memorial was opened to the public on July 30, 2011.