Outgoing Defense Secretary Leon Panetta said the time has come for him to be with his family at his walnut farm to deal "with a different set of nuts," prompting laughter from the audience.