Gypsum fiber composite (GFC) is a kind of building material widely used in interior decoration. Milling is the most commonly used machining process for GFC. Cutting force as an important cutting characteristic parameter has significant influence on the quality of machined surface, power consumption, and tools wear. The tangential force (Fx) and normal force (Fy) were measured and analyzed to find out the effects of milling parameters on these cutting forces. Milling parameters considered were spindle speed, feed rate, and depth of cut. The response surface methodology (RSM) was selected to develop mathematical models and optimize milling parameters. The results showed that with the increase of feed rate and depth of cut, the Fx and Fy increased. But the cutting forces decreased with the increase of spindle speed. The optimization results indicated that high spindle speed, low feed rate, and small depth of cut are preferable for milling of GFC to obtain the best result.