Classic Mode Authentication: It
refers to the integrated windows authentication. You cannot configure
the Forms based authentication if your web application is using Classic
Mode Authentication. You can convert a web application from Classic Mode
Authentication to Claims Based Authentication. However, that can only
be done using PowerShell commands and its an irreversible process.
Claims Based Authentication: SharePoint
2010 is built on Windows Identity Foundation. It enables authentication
from windows as well as non-windows based systems. This also provides
the capability to have multiple authentication in a single URL.