can I add a custom format to the format option in the text panel?

You can add custom format or remove exist in that way:

add_filter('tiny_mce_before_init', function($init_array) {
    $init_array['formats'] = json_encode([
        // add new format to formats
        'h3marked' => [
            'selector' => 'h3',
            'block'    => 'h3',
            'classes'  => 'article-paragraph',
        ],
    ], JSON_THROW_ON_ERROR);

    // remove from that array not needed formats
    $block_formats = [
        'Paragraph=p',
        'Heading 1=h1',
        'Heading 2=h2',
        'Heading 3=h3',
        'Heading 3 marked=h3marked',    // use the new format in select
        'Heading 4=h4',
        'Heading 5=h5',
        'Heading 6=h6',
        'Preformatted=pre',
    ];
    $init_array['block_formats'] = implode(';', $block_formats);

    return $init_array;
});

See more in this post and this tinymce docs

As per here TinyMCE format dropdown no longer showing style previews

Kara had it right, you need to unset the default styles to see the new styles...

unset($init['preview_styles']);

return $settings;

The "classic" TinyMCE editor has two dropdowns: formatselect for paragraph styles and styleselect for character styles - which can also contain paragraph styles, to make it more confusing. The configuration in WordPress by default only shows the format dropdown. If you apply a custom stylesheet to the editor, TinyMCE can use it to pick up the classnames and add them to the style dropdown - but this did not work every time for me.

Since 3.0 you can call add_editor_style() in your functions.php to add a stylesheet to the editor. By default it's editor-style.css in your theme directory. Before 3.0 you have to hook into the mce_css filter to add the URL to your editor stylesheet. This will end up in the content_css TinyMCE configuration value.

To add the style dropdown, the styleselect option must appear in one of the button bar configuration arrays (theme_advanced_buttons[1-4] in TinyMCE, filtered by mce_buttons_[1-4] in WordPress). The list of block formats is controlled by the theme_advanced_blockformats option of TinyMCE, which you can add to the control array in the tiny_mce_before_init filter. If you want to customize the names of the style dropdown (not just your CSS class names), look at the theme_advanced_styles option. You can also use the more advanced style_formats option which gives you more flexibility to define the styles.

The relevant PHP code with all the hooks and default configuration is in wp-admin/includes/post.php, in function wp_tiny_mce(). All together, your setup could look like this:

add_action( 'after_setup_theme', 'wpse3882_after_setup_theme' );
function wpse3882_after_setup_theme()
{
    add_editor_style();
}

add_filter('mce_buttons_2', 'wpse3882_mce_buttons_2');
function wpse3882_mce_buttons_2($buttons)
{
    array_unshift($buttons, 'styleselect');
    return $buttons;
}

add_filter('tiny_mce_before_init', 'wpse3882_tiny_mce_before_init');
function wpse3882_tiny_mce_before_init($settings)
{
    $settings['theme_advanced_blockformats'] = 'p,h1,h2,h3,h4';

    // From http://tinymce.moxiecode.com/examples/example_24.php
    $style_formats = array(
        array('title' => 'Bold text', 'inline' => 'b'),
        array('title' => 'Red text', 'inline' => 'span', 'styles' => array('color' => '#ff0000')),
        array('title' => 'Red header', 'block' => 'h1', 'styles' => array('color' => '#ff0000')),
        array('title' => 'Example 1', 'inline' => 'span', 'classes' => 'example1'),
        array('title' => 'Example 2', 'inline' => 'span', 'classes' => 'example2'),
        array('title' => 'Table styles'),
        array('title' => 'Table row 1', 'selector' => 'tr', 'classes' => 'tablerow1'),
    );
    // Before 3.1 you needed a special trick to send this array to the configuration.
    // See this post history for previous versions.
    $settings['style_formats'] = json_encode( $style_formats );

    return $settings;
}